You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Aditya Auradkar <aa...@linkedin.com.INVALID> on 2015/06/11 19:32:54 UTC
RE: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
I've made two changes to the document:
- Removed the TMR evolution piece since we agreed to retain this.
- Added two new API's to the admin client spec. (Alter and Describe config).
Please review.
Aditya
________________________________________
From: Ashish Singh [asingh@cloudera.com]
Sent: Friday, May 29, 2015 8:36 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
+1 on discussing this on next KIP hangout. I will update KIP-24 before that.
On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <
andrii.biletskyi@stealth.ly> wrote:
> Guys,
>
> I won't be able to attend next meeting. But in the latest patch for KIP-4
> Phase 1
> I didn't even evolve TopicMetadataRequest to v1 since we won't be able
> to change config with AlterTopicRequest, hence with this patch TMR will
> still
> return isr. Taking this into account I think yes - it would be good to fix
> ISR issue,
> although I didn't consider it to be a critical one (isr was part of TMR
> from the very
> beginning and almost no code relies on this piece of request).
>
> Thanks,
> Andrii Biletskyi
>
> On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar <
> aauradkar@linkedin.com.invalid> wrote:
>
> > Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss
> > this during the next hangout?
> >
> > Aditya
> >
> > ________________________________________
> > From: Jun Rao [jun@confluent.io]
> > Sent: Thursday, May 28, 2015 5:32 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> administrative
> > operations (Thread 2)
> >
> > There is a reasonable use case of ISR in KAFKA-2225. Basically, for
> > economical reasons, we may want to let a consumer fetch from a replica in
> > ISR that's in the same zone. In order to support that, it will be
> > convenient to have TMR return the correct ISR for the consumer to choose.
> >
> > So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367
> > (there is some new discussion there on what it takes to fix this). If we
> do
> > that, we can leave TMR unchanged.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar <
> > aauradkar@linkedin.com.invalid> wrote:
> >
> > > Andryii,
> > >
> > > I made a few edits to this document as discussed in the KIP-21 thread.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > >
> > > With these changes. the only difference between
> TopicMetadataResponse_V1
> > > and V0 is the removal of the ISR field. I've altered the KIP with the
> > > assumption that this is a good enough reason by itself to evolve the
> > > request/response protocol. Any concerns there?
> > >
> > > Thanks,
> > > Aditya
> > >
> > > ________________________________________
> > > From: Mayuresh Gharat [gharatmayuresh15@gmail.com]
> > > Sent: Thursday, May 21, 2015 8:29 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > administrative
> > > operations (Thread 2)
> > >
> > > Hi Jun,
> > >
> > > Thanks a lot. I get it now.
> > > Point 4) will actually enable clients to who don't want to create a
> > topic
> > > with default partitions, if it does not exist and then can manually
> > create
> > > the topic with their own configs(#partitions).
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Mayuresh,
> > > >
> > > > The current plan is the following.
> > > >
> > > > 1. Add TMR v1, which still triggers auto topic creation.
> > > > 2. Change the consumer client to TMR v1. Change the producer client
> to
> > > use
> > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to
> > > explicitly
> > > > create the topic with the default server side partitions and
> replicas.
> > > > 3. At some later time after the new clients are released and
> deployed,
> > > > disable auto topic creation in TMR v1. This will make sure consumers
> > > never
> > > > create new topics.
> > > > 4. If needed, we can add a new config in the producer to control
> > whether
> > > > TopicCreateRequest should be issued or not on UnknownTopicException.
> If
> > > > this is disabled and the topic doesn't exist, send will fail and the
> > user
> > > > is expected to create the topic manually.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> > > > gharatmayuresh15@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi,
> > > > > I had a question about TopicMetadata Request.
> > > > > Currently the way it works is :
> > > > >
> > > > > 1) Suppose a topic T1 does not exist.
> > > > > 2) Client wants to produce data to T1 using producer P1.
> > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to
> > kafka.
> > > > > This in turn creates the default number of partition. The number of
> > > > > partitions is a cluster wide config.
> > > > > 4) Same goes for a consumer. If the topic does not exist and new
> > topic
> > > > will
> > > > > be created when the consumer issues TopicMetadata request.
> > > > >
> > > > > Here are 2 use cases where it might not be suited :
> > > > >
> > > > > The auto creation flag for topics is turned ON.
> > > > >
> > > > > a) Some clients might not want to create topic with default number
> of
> > > > > partitions but with lower number of partitions. Currently in a
> > > > multi-tenant
> > > > > environment this is not possible without changing the cluster wide
> > > > default
> > > > > config.
> > > > >
> > > > > b) Some clients might want to just check if the topic exist or not
> > but
> > > > > currently the topic gets created automatically using default number
> > of
> > > > > partitions.
> > > > >
> > > > > Here are some ideas to address this :
> > > > >
> > > > > 1) The way this can be addressed is that TopicMetadata request
> > should
> > > > have
> > > > > a way to specify whether it should only check if the topic exist or
> > > check
> > > > > and create a topic with given number of partitions. If the number
> of
> > > > > partitions is not specified use the default cluster wide config.
> > > > >
> > > > > OR
> > > > >
> > > > > 2) We should only allow TopicMetadata Request to get the metadata
> > > > > explicitly and not allow it to create a new topic. We should have
> > > another
> > > > > Request that takes in config parameters from the user regarding how
> > > > he/she
> > > > > wants the topic to be created. This request can be used if we get
> an
> > > > empty
> > > > > TopicMetadata Response.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > >
> > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > >
> > > > > > For ListTopics, we decided not to add a ListTopics request for
> now
> > > and
> > > > > just
> > > > > > rely on passing in an empty list to TMR. We can revisit this in
> the
> > > > > future
> > > > > > if it becomes an issue.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkoshy.w@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Just had a few minor questions before I join the vote thread.
> > > > > > > Apologies if these have been discussed:
> > > > > > >
> > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just
> > return
> > > > > > > InvalidPartitions instead?
> > > > > > > - AdminClient.listTopics: should we allow listing all
> partitions?
> > > Or
> > > > > > > do you intend for the client to issue listTopics followed by
> > > > > > > describeTopics?
> > > > > > > - On returning future<void> for partition reassignments: do we
> > need
> > > > to
> > > > > > > return any future especially since you have the
> > > > > > > verifyReassignPartitions method? For e.g., what happens if
> the
> > > > > > > controller moves? The get should fail right? The client will
> > then
> > > > > > > need to connect to the new controller and reissue the request
> > but
> > > > > > > will then get ReassignPartitionsInProgress. So in that case
> the
> > > > > > > client any way needs to rely in verifyReassignPartitions.
> > > > > > > - In past hangouts I think either you/Joe were mentioning the
> > need
> > > to
> > > > > > > locate the controller (and possibly other cluster metadata).
> It
> > > > > > > appears we decided to defer this for a future KIP. Correct?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Joel
> > > > > > >
> > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi
> wrote:
> > > > > > > > Guys,
> > > > > > > >
> > > > > > > > I've updated the wiki to reflect all previously discussed
> items
> > > > > > > > (regarding the schema only - this is included to phase 1).
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > > > > >
> > > > > > > > I think we can have the final discussion today (for phase 1)
> > and
> > > > > > > > in case no new remarks I will start the voting thread.
> > > > > > > >
> > > > > > > > With regards to AlterTopicRequest semantics. I agree with
> Jun,
> > > > > > > > and I think it's my bad I focused on "multiple topics in one
> > > > > request".
> > > > > > > > The same situation is possible in ProduceRequest, Fetch,
> > > > > TopicMetadata
> > > > > > > > and we handle it naturally and in the most transparent way -
> we
> > > > > > > > put all separate instructions into a map and thus silently
> > ignore
> > > > > > > > duplicates.
> > > > > > > > This also makes Response part simple too - it's just a map
> > > > > > > Topic->ErrorCode.
> > > > > > > > I think we need to follow the same approach for Alter (and
> > > Create,
> > > > > > > Delete)
> > > > > > > > request. With this we add nothing new in terms of batch
> > requests
> > > > > > > > semantics.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Andrii Biletskyi
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -Regards,
> > > > > Mayuresh R. Gharat
> > > > > (862) 250-7125
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
--
Regards,
Ashish
Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Posted by Andrii Biletskyi <an...@stealth.ly>.
Joel, Aditya,
I believe we don't need another thread to do voting since affected
items are not related to the core proposed changes. I agree people
can explicitly down-vote in case of concerns about the things that
changed.
Thanks,
Andrii Biletskyi
On Fri, Jun 12, 2015 at 8:24 PM, Aditya Auradkar <
aauradkar@linkedin.com.invalid> wrote:
> Aside from the things I mentioned, I don't think there were other changes.
> I'll mark this as adopted since there don't appear to be any concerns.
>
> Aditya
>
> ________________________________________
> From: Joel Koshy [jjkoshy.w@gmail.com]
> Sent: Thursday, June 11, 2015 1:28 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
> operations (Thread 2)
>
> Discussion aside, was there any significant material change besides
> the additions below? If so, then we can avoid the overhead of another
> vote unless someone wants to down-vote these changes.
>
> Joel
>
> On Thu, Jun 11, 2015 at 06:36:36PM +0000, Aditya Auradkar wrote:
> > Andrii,
> >
> > Do we need a new voting thread for this KIP? The last round of votes had
> 3 binding +1's but there's been a fair amount of discussion since then.
> >
> > Aditya
> >
> > ________________________________________
> > From: Aditya Auradkar
> > Sent: Thursday, June 11, 2015 10:32 AM
> > To: dev@kafka.apache.org
> > Subject: RE: [DISCUSS] KIP-4 - Command line and centralized
> administrative operations (Thread 2)
> >
> > I've made two changes to the document:
> > - Removed the TMR evolution piece since we agreed to retain this.
> > - Added two new API's to the admin client spec. (Alter and Describe
> config).
> >
> > Please review.
> >
> > Aditya
> >
> > ________________________________________
> > From: Ashish Singh [asingh@cloudera.com]
> > Sent: Friday, May 29, 2015 8:36 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> administrative operations (Thread 2)
> >
> > +1 on discussing this on next KIP hangout. I will update KIP-24 before
> that.
> >
> > On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <
> > andrii.biletskyi@stealth.ly> wrote:
> >
> > > Guys,
> > >
> > > I won't be able to attend next meeting. But in the latest patch for
> KIP-4
> > > Phase 1
> > > I didn't even evolve TopicMetadataRequest to v1 since we won't be able
> > > to change config with AlterTopicRequest, hence with this patch TMR will
> > > still
> > > return isr. Taking this into account I think yes - it would be good to
> fix
> > > ISR issue,
> > > although I didn't consider it to be a critical one (isr was part of TMR
> > > from the very
> > > beginning and almost no code relies on this piece of request).
> > >
> > > Thanks,
> > > Andrii Biletskyi
> > >
> > > On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar <
> > > aauradkar@linkedin.com.invalid> wrote:
> > >
> > > > Thanks. Perhaps we should leave TMR unchanged for now. Should we
> discuss
> > > > this during the next hangout?
> > > >
> > > > Aditya
> > > >
> > > > ________________________________________
> > > > From: Jun Rao [jun@confluent.io]
> > > > Sent: Thursday, May 28, 2015 5:32 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > > administrative
> > > > operations (Thread 2)
> > > >
> > > > There is a reasonable use case of ISR in KAFKA-2225. Basically, for
> > > > economical reasons, we may want to let a consumer fetch from a
> replica in
> > > > ISR that's in the same zone. In order to support that, it will be
> > > > convenient to have TMR return the correct ISR for the consumer to
> choose.
> > > >
> > > > So, perhaps it's worth fixing the ISR inconsistency issue in
> KAFKA-1367
> > > > (there is some new discussion there on what it takes to fix this).
> If we
> > > do
> > > > that, we can leave TMR unchanged.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar <
> > > > aauradkar@linkedin.com.invalid> wrote:
> > > >
> > > > > Andryii,
> > > > >
> > > > > I made a few edits to this document as discussed in the KIP-21
> thread.
> > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > >
> > > > > With these changes. the only difference between
> > > TopicMetadataResponse_V1
> > > > > and V0 is the removal of the ISR field. I've altered the KIP with
> the
> > > > > assumption that this is a good enough reason by itself to evolve
> the
> > > > > request/response protocol. Any concerns there?
> > > > >
> > > > > Thanks,
> > > > > Aditya
> > > > >
> > > > > ________________________________________
> > > > > From: Mayuresh Gharat [gharatmayuresh15@gmail.com]
> > > > > Sent: Thursday, May 21, 2015 8:29 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > > > administrative
> > > > > operations (Thread 2)
> > > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks a lot. I get it now.
> > > > > Point 4) will actually enable clients to who don't want to create
> a
> > > > topic
> > > > > with default partitions, if it does not exist and then can manually
> > > > create
> > > > > the topic with their own configs(#partitions).
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > Mayuresh,
> > > > > >
> > > > > > The current plan is the following.
> > > > > >
> > > > > > 1. Add TMR v1, which still triggers auto topic creation.
> > > > > > 2. Change the consumer client to TMR v1. Change the producer
> client
> > > to
> > > > > use
> > > > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to
> > > > > explicitly
> > > > > > create the topic with the default server side partitions and
> > > replicas.
> > > > > > 3. At some later time after the new clients are released and
> > > deployed,
> > > > > > disable auto topic creation in TMR v1. This will make sure
> consumers
> > > > > never
> > > > > > create new topics.
> > > > > > 4. If needed, we can add a new config in the producer to control
> > > > whether
> > > > > > TopicCreateRequest should be issued or not on
> UnknownTopicException.
> > > If
> > > > > > this is disabled and the topic doesn't exist, send will fail and
> the
> > > > user
> > > > > > is expected to create the topic manually.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> > > > > > gharatmayuresh15@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I had a question about TopicMetadata Request.
> > > > > > > Currently the way it works is :
> > > > > > >
> > > > > > > 1) Suppose a topic T1 does not exist.
> > > > > > > 2) Client wants to produce data to T1 using producer P1.
> > > > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request
> to
> > > > kafka.
> > > > > > > This in turn creates the default number of partition. The
> number of
> > > > > > > partitions is a cluster wide config.
> > > > > > > 4) Same goes for a consumer. If the topic does not exist and
> new
> > > > topic
> > > > > > will
> > > > > > > be created when the consumer issues TopicMetadata request.
> > > > > > >
> > > > > > > Here are 2 use cases where it might not be suited :
> > > > > > >
> > > > > > > The auto creation flag for topics is turned ON.
> > > > > > >
> > > > > > > a) Some clients might not want to create topic with default
> number
> > > of
> > > > > > > partitions but with lower number of partitions. Currently in a
> > > > > > multi-tenant
> > > > > > > environment this is not possible without changing the cluster
> wide
> > > > > > default
> > > > > > > config.
> > > > > > >
> > > > > > > b) Some clients might want to just check if the topic exist or
> not
> > > > but
> > > > > > > currently the topic gets created automatically using default
> number
> > > > of
> > > > > > > partitions.
> > > > > > >
> > > > > > > Here are some ideas to address this :
> > > > > > >
> > > > > > > 1) The way this can be addressed is that TopicMetadata request
> > > > should
> > > > > > have
> > > > > > > a way to specify whether it should only check if the topic
> exist or
> > > > > check
> > > > > > > and create a topic with given number of partitions. If the
> number
> > > of
> > > > > > > partitions is not specified use the default cluster wide
> config.
> > > > > > >
> > > > > > > OR
> > > > > > >
> > > > > > > 2) We should only allow TopicMetadata Request to get the
> metadata
> > > > > > > explicitly and not allow it to create a new topic. We should
> have
> > > > > another
> > > > > > > Request that takes in config parameters from the user
> regarding how
> > > > > > he/she
> > > > > > > wants the topic to be created. This request can be used if we
> get
> > > an
> > > > > > empty
> > > > > > > TopicMetadata Response.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Mayuresh
> > > > > > >
> > > > > > >
> > > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > > >
> > > > > > > > For ListTopics, we decided not to add a ListTopics request
> for
> > > now
> > > > > and
> > > > > > > just
> > > > > > > > rely on passing in an empty list to TMR. We can revisit this
> in
> > > the
> > > > > > > future
> > > > > > > > if it becomes an issue.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <
> jjkoshy.w@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Just had a few minor questions before I join the vote
> thread.
> > > > > > > > > Apologies if these have been discussed:
> > > > > > > > >
> > > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we
> just
> > > > return
> > > > > > > > > InvalidPartitions instead?
> > > > > > > > > - AdminClient.listTopics: should we allow listing all
> > > partitions?
> > > > > Or
> > > > > > > > > do you intend for the client to issue listTopics
> followed by
> > > > > > > > > describeTopics?
> > > > > > > > > - On returning future<void> for partition reassignments:
> do we
> > > > need
> > > > > > to
> > > > > > > > > return any future especially since you have the
> > > > > > > > > verifyReassignPartitions method? For e.g., what happens
> if
> > > the
> > > > > > > > > controller moves? The get should fail right? The client
> will
> > > > then
> > > > > > > > > need to connect to the new controller and reissue the
> request
> > > > but
> > > > > > > > > will then get ReassignPartitionsInProgress. So in that
> case
> > > the
> > > > > > > > > client any way needs to rely in verifyReassignPartitions.
> > > > > > > > > - In past hangouts I think either you/Joe were mentioning
> the
> > > > need
> > > > > to
> > > > > > > > > locate the controller (and possibly other cluster
> metadata).
> > > It
> > > > > > > > > appears we decided to defer this for a future KIP.
> Correct?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Joel
> > > > > > > > >
> > > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi
> > > wrote:
> > > > > > > > > > Guys,
> > > > > > > > > >
> > > > > > > > > > I've updated the wiki to reflect all previously discussed
> > > items
> > > > > > > > > > (regarding the schema only - this is included to phase
> 1).
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > > > > > > >
> > > > > > > > > > I think we can have the final discussion today (for
> phase 1)
> > > > and
> > > > > > > > > > in case no new remarks I will start the voting thread.
> > > > > > > > > >
> > > > > > > > > > With regards to AlterTopicRequest semantics. I agree with
> > > Jun,
> > > > > > > > > > and I think it's my bad I focused on "multiple topics in
> one
> > > > > > > request".
> > > > > > > > > > The same situation is possible in ProduceRequest, Fetch,
> > > > > > > TopicMetadata
> > > > > > > > > > and we handle it naturally and in the most transparent
> way -
> > > we
> > > > > > > > > > put all separate instructions into a map and thus
> silently
> > > > ignore
> > > > > > > > > > duplicates.
> > > > > > > > > > This also makes Response part simple too - it's just a
> map
> > > > > > > > > Topic->ErrorCode.
> > > > > > > > > > I think we need to follow the same approach for Alter
> (and
> > > > > Create,
> > > > > > > > > Delete)
> > > > > > > > > > request. With this we add nothing new in terms of batch
> > > > requests
> > > > > > > > > > semantics.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Andrii Biletskyi
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -Regards,
> > > > > > > Mayuresh R. Gharat
> > > > > > > (862) 250-7125
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -Regards,
> > > > > Mayuresh R. Gharat
> > > > > (862) 250-7125
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
>
>
RE: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Posted by Aditya Auradkar <aa...@linkedin.com.INVALID>.
Aside from the things I mentioned, I don't think there were other changes. I'll mark this as adopted since there don't appear to be any concerns.
Aditya
________________________________________
From: Joel Koshy [jjkoshy.w@gmail.com]
Sent: Thursday, June 11, 2015 1:28 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
Discussion aside, was there any significant material change besides
the additions below? If so, then we can avoid the overhead of another
vote unless someone wants to down-vote these changes.
Joel
On Thu, Jun 11, 2015 at 06:36:36PM +0000, Aditya Auradkar wrote:
> Andrii,
>
> Do we need a new voting thread for this KIP? The last round of votes had 3 binding +1's but there's been a fair amount of discussion since then.
>
> Aditya
>
> ________________________________________
> From: Aditya Auradkar
> Sent: Thursday, June 11, 2015 10:32 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
>
> I've made two changes to the document:
> - Removed the TMR evolution piece since we agreed to retain this.
> - Added two new API's to the admin client spec. (Alter and Describe config).
>
> Please review.
>
> Aditya
>
> ________________________________________
> From: Ashish Singh [asingh@cloudera.com]
> Sent: Friday, May 29, 2015 8:36 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
>
> +1 on discussing this on next KIP hangout. I will update KIP-24 before that.
>
> On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <
> andrii.biletskyi@stealth.ly> wrote:
>
> > Guys,
> >
> > I won't be able to attend next meeting. But in the latest patch for KIP-4
> > Phase 1
> > I didn't even evolve TopicMetadataRequest to v1 since we won't be able
> > to change config with AlterTopicRequest, hence with this patch TMR will
> > still
> > return isr. Taking this into account I think yes - it would be good to fix
> > ISR issue,
> > although I didn't consider it to be a critical one (isr was part of TMR
> > from the very
> > beginning and almost no code relies on this piece of request).
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar <
> > aauradkar@linkedin.com.invalid> wrote:
> >
> > > Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss
> > > this during the next hangout?
> > >
> > > Aditya
> > >
> > > ________________________________________
> > > From: Jun Rao [jun@confluent.io]
> > > Sent: Thursday, May 28, 2015 5:32 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > administrative
> > > operations (Thread 2)
> > >
> > > There is a reasonable use case of ISR in KAFKA-2225. Basically, for
> > > economical reasons, we may want to let a consumer fetch from a replica in
> > > ISR that's in the same zone. In order to support that, it will be
> > > convenient to have TMR return the correct ISR for the consumer to choose.
> > >
> > > So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367
> > > (there is some new discussion there on what it takes to fix this). If we
> > do
> > > that, we can leave TMR unchanged.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar <
> > > aauradkar@linkedin.com.invalid> wrote:
> > >
> > > > Andryii,
> > > >
> > > > I made a few edits to this document as discussed in the KIP-21 thread.
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > >
> > > > With these changes. the only difference between
> > TopicMetadataResponse_V1
> > > > and V0 is the removal of the ISR field. I've altered the KIP with the
> > > > assumption that this is a good enough reason by itself to evolve the
> > > > request/response protocol. Any concerns there?
> > > >
> > > > Thanks,
> > > > Aditya
> > > >
> > > > ________________________________________
> > > > From: Mayuresh Gharat [gharatmayuresh15@gmail.com]
> > > > Sent: Thursday, May 21, 2015 8:29 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > > administrative
> > > > operations (Thread 2)
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks a lot. I get it now.
> > > > Point 4) will actually enable clients to who don't want to create a
> > > topic
> > > > with default partitions, if it does not exist and then can manually
> > > create
> > > > the topic with their own configs(#partitions).
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Mayuresh,
> > > > >
> > > > > The current plan is the following.
> > > > >
> > > > > 1. Add TMR v1, which still triggers auto topic creation.
> > > > > 2. Change the consumer client to TMR v1. Change the producer client
> > to
> > > > use
> > > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to
> > > > explicitly
> > > > > create the topic with the default server side partitions and
> > replicas.
> > > > > 3. At some later time after the new clients are released and
> > deployed,
> > > > > disable auto topic creation in TMR v1. This will make sure consumers
> > > > never
> > > > > create new topics.
> > > > > 4. If needed, we can add a new config in the producer to control
> > > whether
> > > > > TopicCreateRequest should be issued or not on UnknownTopicException.
> > If
> > > > > this is disabled and the topic doesn't exist, send will fail and the
> > > user
> > > > > is expected to create the topic manually.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> > > > > gharatmayuresh15@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I had a question about TopicMetadata Request.
> > > > > > Currently the way it works is :
> > > > > >
> > > > > > 1) Suppose a topic T1 does not exist.
> > > > > > 2) Client wants to produce data to T1 using producer P1.
> > > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to
> > > kafka.
> > > > > > This in turn creates the default number of partition. The number of
> > > > > > partitions is a cluster wide config.
> > > > > > 4) Same goes for a consumer. If the topic does not exist and new
> > > topic
> > > > > will
> > > > > > be created when the consumer issues TopicMetadata request.
> > > > > >
> > > > > > Here are 2 use cases where it might not be suited :
> > > > > >
> > > > > > The auto creation flag for topics is turned ON.
> > > > > >
> > > > > > a) Some clients might not want to create topic with default number
> > of
> > > > > > partitions but with lower number of partitions. Currently in a
> > > > > multi-tenant
> > > > > > environment this is not possible without changing the cluster wide
> > > > > default
> > > > > > config.
> > > > > >
> > > > > > b) Some clients might want to just check if the topic exist or not
> > > but
> > > > > > currently the topic gets created automatically using default number
> > > of
> > > > > > partitions.
> > > > > >
> > > > > > Here are some ideas to address this :
> > > > > >
> > > > > > 1) The way this can be addressed is that TopicMetadata request
> > > should
> > > > > have
> > > > > > a way to specify whether it should only check if the topic exist or
> > > > check
> > > > > > and create a topic with given number of partitions. If the number
> > of
> > > > > > partitions is not specified use the default cluster wide config.
> > > > > >
> > > > > > OR
> > > > > >
> > > > > > 2) We should only allow TopicMetadata Request to get the metadata
> > > > > > explicitly and not allow it to create a new topic. We should have
> > > > another
> > > > > > Request that takes in config parameters from the user regarding how
> > > > > he/she
> > > > > > wants the topic to be created. This request can be used if we get
> > an
> > > > > empty
> > > > > > TopicMetadata Response.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mayuresh
> > > > > >
> > > > > >
> > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > >
> > > > > > > For ListTopics, we decided not to add a ListTopics request for
> > now
> > > > and
> > > > > > just
> > > > > > > rely on passing in an empty list to TMR. We can revisit this in
> > the
> > > > > > future
> > > > > > > if it becomes an issue.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkoshy.w@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Just had a few minor questions before I join the vote thread.
> > > > > > > > Apologies if these have been discussed:
> > > > > > > >
> > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just
> > > return
> > > > > > > > InvalidPartitions instead?
> > > > > > > > - AdminClient.listTopics: should we allow listing all
> > partitions?
> > > > Or
> > > > > > > > do you intend for the client to issue listTopics followed by
> > > > > > > > describeTopics?
> > > > > > > > - On returning future<void> for partition reassignments: do we
> > > need
> > > > > to
> > > > > > > > return any future especially since you have the
> > > > > > > > verifyReassignPartitions method? For e.g., what happens if
> > the
> > > > > > > > controller moves? The get should fail right? The client will
> > > then
> > > > > > > > need to connect to the new controller and reissue the request
> > > but
> > > > > > > > will then get ReassignPartitionsInProgress. So in that case
> > the
> > > > > > > > client any way needs to rely in verifyReassignPartitions.
> > > > > > > > - In past hangouts I think either you/Joe were mentioning the
> > > need
> > > > to
> > > > > > > > locate the controller (and possibly other cluster metadata).
> > It
> > > > > > > > appears we decided to defer this for a future KIP. Correct?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Joel
> > > > > > > >
> > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi
> > wrote:
> > > > > > > > > Guys,
> > > > > > > > >
> > > > > > > > > I've updated the wiki to reflect all previously discussed
> > items
> > > > > > > > > (regarding the schema only - this is included to phase 1).
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > > > > > >
> > > > > > > > > I think we can have the final discussion today (for phase 1)
> > > and
> > > > > > > > > in case no new remarks I will start the voting thread.
> > > > > > > > >
> > > > > > > > > With regards to AlterTopicRequest semantics. I agree with
> > Jun,
> > > > > > > > > and I think it's my bad I focused on "multiple topics in one
> > > > > > request".
> > > > > > > > > The same situation is possible in ProduceRequest, Fetch,
> > > > > > TopicMetadata
> > > > > > > > > and we handle it naturally and in the most transparent way -
> > we
> > > > > > > > > put all separate instructions into a map and thus silently
> > > ignore
> > > > > > > > > duplicates.
> > > > > > > > > This also makes Response part simple too - it's just a map
> > > > > > > > Topic->ErrorCode.
> > > > > > > > > I think we need to follow the same approach for Alter (and
> > > > Create,
> > > > > > > > Delete)
> > > > > > > > > request. With this we add nothing new in terms of batch
> > > requests
> > > > > > > > > semantics.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Andrii Biletskyi
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -Regards,
> > > > > > Mayuresh R. Gharat
> > > > > > (862) 250-7125
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -Regards,
> > > > Mayuresh R. Gharat
> > > > (862) 250-7125
> > > >
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Posted by Joel Koshy <jj...@gmail.com>.
Discussion aside, was there any significant material change besides
the additions below? If so, then we can avoid the overhead of another
vote unless someone wants to down-vote these changes.
Joel
On Thu, Jun 11, 2015 at 06:36:36PM +0000, Aditya Auradkar wrote:
> Andrii,
>
> Do we need a new voting thread for this KIP? The last round of votes had 3 binding +1's but there's been a fair amount of discussion since then.
>
> Aditya
>
> ________________________________________
> From: Aditya Auradkar
> Sent: Thursday, June 11, 2015 10:32 AM
> To: dev@kafka.apache.org
> Subject: RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
>
> I've made two changes to the document:
> - Removed the TMR evolution piece since we agreed to retain this.
> - Added two new API's to the admin client spec. (Alter and Describe config).
>
> Please review.
>
> Aditya
>
> ________________________________________
> From: Ashish Singh [asingh@cloudera.com]
> Sent: Friday, May 29, 2015 8:36 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
>
> +1 on discussing this on next KIP hangout. I will update KIP-24 before that.
>
> On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <
> andrii.biletskyi@stealth.ly> wrote:
>
> > Guys,
> >
> > I won't be able to attend next meeting. But in the latest patch for KIP-4
> > Phase 1
> > I didn't even evolve TopicMetadataRequest to v1 since we won't be able
> > to change config with AlterTopicRequest, hence with this patch TMR will
> > still
> > return isr. Taking this into account I think yes - it would be good to fix
> > ISR issue,
> > although I didn't consider it to be a critical one (isr was part of TMR
> > from the very
> > beginning and almost no code relies on this piece of request).
> >
> > Thanks,
> > Andrii Biletskyi
> >
> > On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar <
> > aauradkar@linkedin.com.invalid> wrote:
> >
> > > Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss
> > > this during the next hangout?
> > >
> > > Aditya
> > >
> > > ________________________________________
> > > From: Jun Rao [jun@confluent.io]
> > > Sent: Thursday, May 28, 2015 5:32 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > administrative
> > > operations (Thread 2)
> > >
> > > There is a reasonable use case of ISR in KAFKA-2225. Basically, for
> > > economical reasons, we may want to let a consumer fetch from a replica in
> > > ISR that's in the same zone. In order to support that, it will be
> > > convenient to have TMR return the correct ISR for the consumer to choose.
> > >
> > > So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367
> > > (there is some new discussion there on what it takes to fix this). If we
> > do
> > > that, we can leave TMR unchanged.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar <
> > > aauradkar@linkedin.com.invalid> wrote:
> > >
> > > > Andryii,
> > > >
> > > > I made a few edits to this document as discussed in the KIP-21 thread.
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > >
> > > > With these changes. the only difference between
> > TopicMetadataResponse_V1
> > > > and V0 is the removal of the ISR field. I've altered the KIP with the
> > > > assumption that this is a good enough reason by itself to evolve the
> > > > request/response protocol. Any concerns there?
> > > >
> > > > Thanks,
> > > > Aditya
> > > >
> > > > ________________________________________
> > > > From: Mayuresh Gharat [gharatmayuresh15@gmail.com]
> > > > Sent: Thursday, May 21, 2015 8:29 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > > administrative
> > > > operations (Thread 2)
> > > >
> > > > Hi Jun,
> > > >
> > > > Thanks a lot. I get it now.
> > > > Point 4) will actually enable clients to who don't want to create a
> > > topic
> > > > with default partitions, if it does not exist and then can manually
> > > create
> > > > the topic with their own configs(#partitions).
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > Mayuresh,
> > > > >
> > > > > The current plan is the following.
> > > > >
> > > > > 1. Add TMR v1, which still triggers auto topic creation.
> > > > > 2. Change the consumer client to TMR v1. Change the producer client
> > to
> > > > use
> > > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to
> > > > explicitly
> > > > > create the topic with the default server side partitions and
> > replicas.
> > > > > 3. At some later time after the new clients are released and
> > deployed,
> > > > > disable auto topic creation in TMR v1. This will make sure consumers
> > > > never
> > > > > create new topics.
> > > > > 4. If needed, we can add a new config in the producer to control
> > > whether
> > > > > TopicCreateRequest should be issued or not on UnknownTopicException.
> > If
> > > > > this is disabled and the topic doesn't exist, send will fail and the
> > > user
> > > > > is expected to create the topic manually.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> > > > > gharatmayuresh15@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I had a question about TopicMetadata Request.
> > > > > > Currently the way it works is :
> > > > > >
> > > > > > 1) Suppose a topic T1 does not exist.
> > > > > > 2) Client wants to produce data to T1 using producer P1.
> > > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to
> > > kafka.
> > > > > > This in turn creates the default number of partition. The number of
> > > > > > partitions is a cluster wide config.
> > > > > > 4) Same goes for a consumer. If the topic does not exist and new
> > > topic
> > > > > will
> > > > > > be created when the consumer issues TopicMetadata request.
> > > > > >
> > > > > > Here are 2 use cases where it might not be suited :
> > > > > >
> > > > > > The auto creation flag for topics is turned ON.
> > > > > >
> > > > > > a) Some clients might not want to create topic with default number
> > of
> > > > > > partitions but with lower number of partitions. Currently in a
> > > > > multi-tenant
> > > > > > environment this is not possible without changing the cluster wide
> > > > > default
> > > > > > config.
> > > > > >
> > > > > > b) Some clients might want to just check if the topic exist or not
> > > but
> > > > > > currently the topic gets created automatically using default number
> > > of
> > > > > > partitions.
> > > > > >
> > > > > > Here are some ideas to address this :
> > > > > >
> > > > > > 1) The way this can be addressed is that TopicMetadata request
> > > should
> > > > > have
> > > > > > a way to specify whether it should only check if the topic exist or
> > > > check
> > > > > > and create a topic with given number of partitions. If the number
> > of
> > > > > > partitions is not specified use the default cluster wide config.
> > > > > >
> > > > > > OR
> > > > > >
> > > > > > 2) We should only allow TopicMetadata Request to get the metadata
> > > > > > explicitly and not allow it to create a new topic. We should have
> > > > another
> > > > > > Request that takes in config parameters from the user regarding how
> > > > > he/she
> > > > > > wants the topic to be created. This request can be used if we get
> > an
> > > > > empty
> > > > > > TopicMetadata Response.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mayuresh
> > > > > >
> > > > > >
> > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <ju...@confluent.io>
> > wrote:
> > > > > >
> > > > > > > For ListTopics, we decided not to add a ListTopics request for
> > now
> > > > and
> > > > > > just
> > > > > > > rely on passing in an empty list to TMR. We can revisit this in
> > the
> > > > > > future
> > > > > > > if it becomes an issue.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkoshy.w@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Just had a few minor questions before I join the vote thread.
> > > > > > > > Apologies if these have been discussed:
> > > > > > > >
> > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just
> > > return
> > > > > > > > InvalidPartitions instead?
> > > > > > > > - AdminClient.listTopics: should we allow listing all
> > partitions?
> > > > Or
> > > > > > > > do you intend for the client to issue listTopics followed by
> > > > > > > > describeTopics?
> > > > > > > > - On returning future<void> for partition reassignments: do we
> > > need
> > > > > to
> > > > > > > > return any future especially since you have the
> > > > > > > > verifyReassignPartitions method? For e.g., what happens if
> > the
> > > > > > > > controller moves? The get should fail right? The client will
> > > then
> > > > > > > > need to connect to the new controller and reissue the request
> > > but
> > > > > > > > will then get ReassignPartitionsInProgress. So in that case
> > the
> > > > > > > > client any way needs to rely in verifyReassignPartitions.
> > > > > > > > - In past hangouts I think either you/Joe were mentioning the
> > > need
> > > > to
> > > > > > > > locate the controller (and possibly other cluster metadata).
> > It
> > > > > > > > appears we decided to defer this for a future KIP. Correct?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Joel
> > > > > > > >
> > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi
> > wrote:
> > > > > > > > > Guys,
> > > > > > > > >
> > > > > > > > > I've updated the wiki to reflect all previously discussed
> > items
> > > > > > > > > (regarding the schema only - this is included to phase 1).
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > > > > > >
> > > > > > > > > I think we can have the final discussion today (for phase 1)
> > > and
> > > > > > > > > in case no new remarks I will start the voting thread.
> > > > > > > > >
> > > > > > > > > With regards to AlterTopicRequest semantics. I agree with
> > Jun,
> > > > > > > > > and I think it's my bad I focused on "multiple topics in one
> > > > > > request".
> > > > > > > > > The same situation is possible in ProduceRequest, Fetch,
> > > > > > TopicMetadata
> > > > > > > > > and we handle it naturally and in the most transparent way -
> > we
> > > > > > > > > put all separate instructions into a map and thus silently
> > > ignore
> > > > > > > > > duplicates.
> > > > > > > > > This also makes Response part simple too - it's just a map
> > > > > > > > Topic->ErrorCode.
> > > > > > > > > I think we need to follow the same approach for Alter (and
> > > > Create,
> > > > > > > > Delete)
> > > > > > > > > request. With this we add nothing new in terms of batch
> > > requests
> > > > > > > > > semantics.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Andrii Biletskyi
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -Regards,
> > > > > > Mayuresh R. Gharat
> > > > > > (862) 250-7125
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -Regards,
> > > > Mayuresh R. Gharat
> > > > (862) 250-7125
> > > >
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
RE: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Posted by Aditya Auradkar <aa...@linkedin.com.INVALID>.
Andrii,
Do we need a new voting thread for this KIP? The last round of votes had 3 binding +1's but there's been a fair amount of discussion since then.
Aditya
________________________________________
From: Aditya Auradkar
Sent: Thursday, June 11, 2015 10:32 AM
To: dev@kafka.apache.org
Subject: RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
I've made two changes to the document:
- Removed the TMR evolution piece since we agreed to retain this.
- Added two new API's to the admin client spec. (Alter and Describe config).
Please review.
Aditya
________________________________________
From: Ashish Singh [asingh@cloudera.com]
Sent: Friday, May 29, 2015 8:36 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)
+1 on discussing this on next KIP hangout. I will update KIP-24 before that.
On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <
andrii.biletskyi@stealth.ly> wrote:
> Guys,
>
> I won't be able to attend next meeting. But in the latest patch for KIP-4
> Phase 1
> I didn't even evolve TopicMetadataRequest to v1 since we won't be able
> to change config with AlterTopicRequest, hence with this patch TMR will
> still
> return isr. Taking this into account I think yes - it would be good to fix
> ISR issue,
> although I didn't consider it to be a critical one (isr was part of TMR
> from the very
> beginning and almost no code relies on this piece of request).
>
> Thanks,
> Andrii Biletskyi
>
> On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar <
> aauradkar@linkedin.com.invalid> wrote:
>
> > Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss
> > this during the next hangout?
> >
> > Aditya
> >
> > ________________________________________
> > From: Jun Rao [jun@confluent.io]
> > Sent: Thursday, May 28, 2015 5:32 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> administrative
> > operations (Thread 2)
> >
> > There is a reasonable use case of ISR in KAFKA-2225. Basically, for
> > economical reasons, we may want to let a consumer fetch from a replica in
> > ISR that's in the same zone. In order to support that, it will be
> > convenient to have TMR return the correct ISR for the consumer to choose.
> >
> > So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367
> > (there is some new discussion there on what it takes to fix this). If we
> do
> > that, we can leave TMR unchanged.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar <
> > aauradkar@linkedin.com.invalid> wrote:
> >
> > > Andryii,
> > >
> > > I made a few edits to this document as discussed in the KIP-21 thread.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > >
> > > With these changes. the only difference between
> TopicMetadataResponse_V1
> > > and V0 is the removal of the ISR field. I've altered the KIP with the
> > > assumption that this is a good enough reason by itself to evolve the
> > > request/response protocol. Any concerns there?
> > >
> > > Thanks,
> > > Aditya
> > >
> > > ________________________________________
> > > From: Mayuresh Gharat [gharatmayuresh15@gmail.com]
> > > Sent: Thursday, May 21, 2015 8:29 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized
> > administrative
> > > operations (Thread 2)
> > >
> > > Hi Jun,
> > >
> > > Thanks a lot. I get it now.
> > > Point 4) will actually enable clients to who don't want to create a
> > topic
> > > with default partitions, if it does not exist and then can manually
> > create
> > > the topic with their own configs(#partitions).
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Mayuresh,
> > > >
> > > > The current plan is the following.
> > > >
> > > > 1. Add TMR v1, which still triggers auto topic creation.
> > > > 2. Change the consumer client to TMR v1. Change the producer client
> to
> > > use
> > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to
> > > explicitly
> > > > create the topic with the default server side partitions and
> replicas.
> > > > 3. At some later time after the new clients are released and
> deployed,
> > > > disable auto topic creation in TMR v1. This will make sure consumers
> > > never
> > > > create new topics.
> > > > 4. If needed, we can add a new config in the producer to control
> > whether
> > > > TopicCreateRequest should be issued or not on UnknownTopicException.
> If
> > > > this is disabled and the topic doesn't exist, send will fail and the
> > user
> > > > is expected to create the topic manually.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <
> > > > gharatmayuresh15@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi,
> > > > > I had a question about TopicMetadata Request.
> > > > > Currently the way it works is :
> > > > >
> > > > > 1) Suppose a topic T1 does not exist.
> > > > > 2) Client wants to produce data to T1 using producer P1.
> > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to
> > kafka.
> > > > > This in turn creates the default number of partition. The number of
> > > > > partitions is a cluster wide config.
> > > > > 4) Same goes for a consumer. If the topic does not exist and new
> > topic
> > > > will
> > > > > be created when the consumer issues TopicMetadata request.
> > > > >
> > > > > Here are 2 use cases where it might not be suited :
> > > > >
> > > > > The auto creation flag for topics is turned ON.
> > > > >
> > > > > a) Some clients might not want to create topic with default number
> of
> > > > > partitions but with lower number of partitions. Currently in a
> > > > multi-tenant
> > > > > environment this is not possible without changing the cluster wide
> > > > default
> > > > > config.
> > > > >
> > > > > b) Some clients might want to just check if the topic exist or not
> > but
> > > > > currently the topic gets created automatically using default number
> > of
> > > > > partitions.
> > > > >
> > > > > Here are some ideas to address this :
> > > > >
> > > > > 1) The way this can be addressed is that TopicMetadata request
> > should
> > > > have
> > > > > a way to specify whether it should only check if the topic exist or
> > > check
> > > > > and create a topic with given number of partitions. If the number
> of
> > > > > partitions is not specified use the default cluster wide config.
> > > > >
> > > > > OR
> > > > >
> > > > > 2) We should only allow TopicMetadata Request to get the metadata
> > > > > explicitly and not allow it to create a new topic. We should have
> > > another
> > > > > Request that takes in config parameters from the user regarding how
> > > > he/she
> > > > > wants the topic to be created. This request can be used if we get
> an
> > > > empty
> > > > > TopicMetadata Response.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > >
> > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <ju...@confluent.io>
> wrote:
> > > > >
> > > > > > For ListTopics, we decided not to add a ListTopics request for
> now
> > > and
> > > > > just
> > > > > > rely on passing in an empty list to TMR. We can revisit this in
> the
> > > > > future
> > > > > > if it becomes an issue.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkoshy.w@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Just had a few minor questions before I join the vote thread.
> > > > > > > Apologies if these have been discussed:
> > > > > > >
> > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just
> > return
> > > > > > > InvalidPartitions instead?
> > > > > > > - AdminClient.listTopics: should we allow listing all
> partitions?
> > > Or
> > > > > > > do you intend for the client to issue listTopics followed by
> > > > > > > describeTopics?
> > > > > > > - On returning future<void> for partition reassignments: do we
> > need
> > > > to
> > > > > > > return any future especially since you have the
> > > > > > > verifyReassignPartitions method? For e.g., what happens if
> the
> > > > > > > controller moves? The get should fail right? The client will
> > then
> > > > > > > need to connect to the new controller and reissue the request
> > but
> > > > > > > will then get ReassignPartitionsInProgress. So in that case
> the
> > > > > > > client any way needs to rely in verifyReassignPartitions.
> > > > > > > - In past hangouts I think either you/Joe were mentioning the
> > need
> > > to
> > > > > > > locate the controller (and possibly other cluster metadata).
> It
> > > > > > > appears we decided to defer this for a future KIP. Correct?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Joel
> > > > > > >
> > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi
> wrote:
> > > > > > > > Guys,
> > > > > > > >
> > > > > > > > I've updated the wiki to reflect all previously discussed
> items
> > > > > > > > (regarding the schema only - this is included to phase 1).
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
> > > > > > > >
> > > > > > > > I think we can have the final discussion today (for phase 1)
> > and
> > > > > > > > in case no new remarks I will start the voting thread.
> > > > > > > >
> > > > > > > > With regards to AlterTopicRequest semantics. I agree with
> Jun,
> > > > > > > > and I think it's my bad I focused on "multiple topics in one
> > > > > request".
> > > > > > > > The same situation is possible in ProduceRequest, Fetch,
> > > > > TopicMetadata
> > > > > > > > and we handle it naturally and in the most transparent way -
> we
> > > > > > > > put all separate instructions into a map and thus silently
> > ignore
> > > > > > > > duplicates.
> > > > > > > > This also makes Response part simple too - it's just a map
> > > > > > > Topic->ErrorCode.
> > > > > > > > I think we need to follow the same approach for Alter (and
> > > Create,
> > > > > > > Delete)
> > > > > > > > request. With this we add nothing new in terms of batch
> > requests
> > > > > > > > semantics.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Andrii Biletskyi
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -Regards,
> > > > > Mayuresh R. Gharat
> > > > > (862) 250-7125
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
--
Regards,
Ashish