You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Andrew Schofield <an...@outlook.com> on 2024/03/17 21:21:35 UTC
[VOTE] KIP-932: Queues for Kafka
Hi,
I’ve been working to complete KIP-932 over the past few months and discussions have quietened down.
I’d like to open the voting for KIP-932:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
Thanks,
Andrew
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Andrew Schofield <an...@live.com>.
Hi,
KIP-932 has been approved.
+5 (binding) - Edoardo Comar, Jun Rao, Satish Duggana, Manikumar Reddy,
David Jacot
+4 (non-binding) - Lianet Magrans, Apoorv Mittal, Kenneth Eversole,
Omnia Ibrahim
Thanks very much to everyone who put so much effort into reviewing this
complicated KIP. The aim is to have an early access release in Apache
Kafka 4.0.
Thanks,
Andrew
> On 16 May 2024, at 15:45, Omnia Ibrahim <o....@gmail.com> wrote:
>
> Thanks for the KIP Andrew +1 none binding from me
>
>> On 16 May 2024, at 14:23, David Jacot <dj...@confluent.io.INVALID> wrote:
>>
>> Hi Andrew,
>>
>> Thanks for the KIP! This is really exciting! +1 (binding) from me.
>>
>> One note regarding the partition assignor interface changes that you
>> proposed, it would be great to get the changes in 3.8 in order to not break
>> the API of KIP-848 after the preview.
>>
>> Best,
>> David
>>
>> On Wed, May 15, 2024 at 10:37 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>>
>>> Hi, Andrew,
>>>
>>> Thanks for the update. Should we mark whether those metrics are
>>> standard/required for KIP-714?
>>>
>>> Jun
>>>
>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
>>> andrew_schofield@live.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I have made a small update to the KIP as a result of testing the new
>>>> share consumer with client telemetry (KIP-714).
>>>>
>>>> I’ve added telemetry metric names to the table of client metrics and
>>>> also updated the metric group names so that the resulting client metrics
>>>> sent to the broker have consistent names.
>>>>
>>>> Thanks,
>>>> Andrew
>>>>
>>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
>>>>>
>>>>> Hi Andrew,
>>>>>
>>>>> Thanks for the KIP. Great write-up!
>>>>>
>>>>> +1 (binding)
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
>>> satish.duggana@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>> Thanks for the nice KIP, it will allow other messaging use cases to be
>>>>>> onboarded to Kafka.
>>>>>>
>>>>>> +1 from me.
>>>>>>
>>>>>> Satish.
>>>>>>
>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
>>> wrote:
>>>>>>>
>>>>>>> Hi, Andrew,
>>>>>>>
>>>>>>> Thanks for the KIP. +1
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
>>> edoardlists@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Andrew,
>>>>>>>>
>>>>>>>> +1 (binding)
>>>>>>>>
>>>>>>>> Edo
>>>>>>>>
>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
>>>>>>>> <ke...@cloudflare.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>> Hi Andrew
>>>>>>>>>
>>>>>>>>> + 1 (Non-Binding)
>>>>>>>>>
>>>>>>>>> This will be great addition to Kafka
>>>>>>>>>
>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
>>>> apoorvmittal10@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Andrew,
>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
>>>>>>>> addition
>>>>>>>>>> to the Kafka, excited to see the KIP.
>>>>>>>>>>
>>>>>>>>>> + 1 (Non-Binding)
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Apoorv Mittal
>>>>>>>>>> +44 7721681581
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few months
>>> and
>>>>>>>>>>> discussions have quietened down.
>>>>>>>>>>>
>>>>>>>>>>> I’d like to open the voting for KIP-932:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>
>>>>
>>>>
>>>
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Omnia Ibrahim <o....@gmail.com>.
Thanks for the KIP Andrew +1 none binding from me
> On 16 May 2024, at 14:23, David Jacot <dj...@confluent.io.INVALID> wrote:
>
> Hi Andrew,
>
> Thanks for the KIP! This is really exciting! +1 (binding) from me.
>
> One note regarding the partition assignor interface changes that you
> proposed, it would be great to get the changes in 3.8 in order to not break
> the API of KIP-848 after the preview.
>
> Best,
> David
>
> On Wed, May 15, 2024 at 10:37 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
>> Hi, Andrew,
>>
>> Thanks for the update. Should we mark whether those metrics are
>> standard/required for KIP-714?
>>
>> Jun
>>
>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
>> andrew_schofield@live.com>
>> wrote:
>>
>>> Hi,
>>> I have made a small update to the KIP as a result of testing the new
>>> share consumer with client telemetry (KIP-714).
>>>
>>> I’ve added telemetry metric names to the table of client metrics and
>>> also updated the metric group names so that the resulting client metrics
>>> sent to the broker have consistent names.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> Thanks for the KIP. Great write-up!
>>>>
>>>> +1 (binding)
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
>> satish.duggana@gmail.com>
>>> wrote:
>>>>>
>>>>> Hi Andrew,
>>>>> Thanks for the nice KIP, it will allow other messaging use cases to be
>>>>> onboarded to Kafka.
>>>>>
>>>>> +1 from me.
>>>>>
>>>>> Satish.
>>>>>
>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
>> wrote:
>>>>>>
>>>>>> Hi, Andrew,
>>>>>>
>>>>>> Thanks for the KIP. +1
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
>> edoardlists@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Andrew,
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> Edo
>>>>>>>
>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
>>>>>>> <ke...@cloudflare.com.invalid> wrote:
>>>>>>>>
>>>>>>>> Hi Andrew
>>>>>>>>
>>>>>>>> + 1 (Non-Binding)
>>>>>>>>
>>>>>>>> This will be great addition to Kafka
>>>>>>>>
>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
>>> apoorvmittal10@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Andrew,
>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
>>>>>>> addition
>>>>>>>>> to the Kafka, excited to see the KIP.
>>>>>>>>>
>>>>>>>>> + 1 (Non-Binding)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Apoorv Mittal
>>>>>>>>> +44 7721681581
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I’ve been working to complete KIP-932 over the past few months
>> and
>>>>>>>>>> discussions have quietened down.
>>>>>>>>>>
>>>>>>>>>> I’d like to open the voting for KIP-932:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>
>>>
>>>
>>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by David Jacot <dj...@confluent.io.INVALID>.
Hi Andrew,
Thanks for the KIP! This is really exciting! +1 (binding) from me.
One note regarding the partition assignor interface changes that you
proposed, it would be great to get the changes in 3.8 in order to not break
the API of KIP-848 after the preview.
Best,
David
On Wed, May 15, 2024 at 10:37 PM Jun Rao <ju...@confluent.io.invalid> wrote:
> Hi, Andrew,
>
> Thanks for the update. Should we mark whether those metrics are
> standard/required for KIP-714?
>
> Jun
>
> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> andrew_schofield@live.com>
> wrote:
>
> > Hi,
> > I have made a small update to the KIP as a result of testing the new
> > share consumer with client telemetry (KIP-714).
> >
> > I’ve added telemetry metric names to the table of client metrics and
> > also updated the metric group names so that the resulting client metrics
> > sent to the broker have consistent names.
> >
> > Thanks,
> > Andrew
> >
> > > On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
> > >
> > > Hi Andrew,
> > >
> > > Thanks for the KIP. Great write-up!
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > >
> > > On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> satish.duggana@gmail.com>
> > wrote:
> > >>
> > >> Hi Andrew,
> > >> Thanks for the nice KIP, it will allow other messaging use cases to be
> > >> onboarded to Kafka.
> > >>
> > >> +1 from me.
> > >>
> > >> Satish.
> > >>
> > >> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
> wrote:
> > >>>
> > >>> Hi, Andrew,
> > >>>
> > >>> Thanks for the KIP. +1
> > >>>
> > >>> Jun
> > >>>
> > >>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> edoardlists@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Thanks Andrew,
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>> Edo
> > >>>>
> > >>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> > >>>> <ke...@cloudflare.com.invalid> wrote:
> > >>>>>
> > >>>>> Hi Andrew
> > >>>>>
> > >>>>> + 1 (Non-Binding)
> > >>>>>
> > >>>>> This will be great addition to Kafka
> > >>>>>
> > >>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> > apoorvmittal10@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi Andrew,
> > >>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
> > >>>> addition
> > >>>>>> to the Kafka, excited to see the KIP.
> > >>>>>>
> > >>>>>> + 1 (Non-Binding)
> > >>>>>>
> > >>>>>> Regards,
> > >>>>>> Apoorv Mittal
> > >>>>>> +44 7721681581
> > >>>>>>
> > >>>>>>
> > >>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > >>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>
> > >>>>>>> Hi,
> > >>>>>>> I’ve been working to complete KIP-932 over the past few months
> and
> > >>>>>>> discussions have quietened down.
> > >>>>>>>
> > >>>>>>> I’d like to open the voting for KIP-932:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Andrew
> > >>>>>>
> > >>>>
> >
> >
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Andrew Schofield <an...@outlook.com>.
Hi Jun,
Yes, sorry for the confusion. I should have added it before I sent my email
but have done so now. The PR is under review.
Thanks,
Andrew
________________________________________
From: Jun Rao <ju...@confluent.io.INVALID>
Sent: 11 July 2025 18:29
To: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: Re: [VOTE] KIP-932: Queues for Kafka
Hi, Andrew,
Thanks for the updates. Looks reasonable to me.
I don't see ShareConsumer.acknowledge(String, int, long, AcknowledgeType)
in the KIP or the code. There are only the following two.
void acknowledge(ConsumerRecord<K, V> record);
void acknowledge(ConsumerRecord<K, V> record, AcknowledgeType type);
Thanks,
Jun
On Fri, Jul 11, 2025 at 2:30 AM Andrew Schofield <
andrew_schofield_jira@outlook.com> wrote:
> Hi,
> As we've written the code for KIP-932, we've made some small changes to
> the KIP
> to correct mistakes, align with other accepted KIPs and fix some omissions
> picked
> up during PR reviews. We also have experience of people writing
> applications
> against the ShareConsumer interface and have corrected a couple of
> problems.
>
> * Using share.version=1 as the switch to enable share groups. This aligns
> with group.version=1 for KIP-848 and streams.version=1 for KIP-1071.
> * Aligned with KIP-1043 by using a common GroupState and removing
> Admin.listShareGroups (because Admin.listGroups handles all group types).
> * Corrected the topic authorizations on admin requests.
> * Added SHARE_SESSION_LIMIT_REACHED error code to give proper
> consumer error handling when the limit is reached.
> * Adjusted ShareFetch RPC request to remove PartitionMaxBytes and
> add BatchSize to improve efficiency of fetching and state management.
> * Changed the rules for explicit acknowledgement because, as originally
> written, it was simple to write an application that went into an infinite
> loop.
> Now, when using explicit acknowledgement, it is mandatory to acknowledge
> every single record, which allows an exception to be thrown if the
> application
> does not do this properly.
> * Added a method overload with signature
> ShareConsumer.acknowledge(String, int, long, AcknowledgeType) to go with
> ShareConsumer.acknowledge(ConsumerRecord, AcknowledgeType).
> This is because the KIP said it would be possible to choose the acknowledge
> type in the event of RecordDeserializationException. However, because the
> application does not have the ConsumerRecord instance when the
> deserialization
> fails, but it does have topic name, partition and offset, adding an
> overloaded
> method enables the application to choose the acknowledge type.
> * The SimpleAssignor no longer uses hashing as part of making assignments.
> In practice, this hindered the assignment computation. SimpleAssignor
> achieves
> the balance and stickiness described previously, just without hashing.
> * Added
> org.apache.kafka.coordinator.group.api.assignor.GroupSpec.isPartitionAssignable
> so that partitions which are in the process of being initialized in the
> share coordinator
> can be omitted from assignment computation.
> * Removed the ShareGroupPartitionMetadata record type (KIP-1101).
> * Added timestamps to ShareSnapshot records so it is possible to manage
> pruning of records correctly.
> * Adjusted some of the config defaults and limits such as maximum for lock
> duration which didn't make sense.
>
> KIP-932 mentions some future work and I have started to create new KIPs
> for this.
>
> Thanks,
> Andrew
> ________________________________________
> From: Jun Rao <ju...@confluent.io.INVALID>
> Sent: 15 August 2024 19:07
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: Re: [VOTE] KIP-932: Queues for Kafka
>
> Hi, Andrew,
>
> Thanks for the clarification. Sounds good.
>
> Jun
>
> On Thu, Aug 15, 2024 at 2:53 AM Andrew Schofield <
> andrew_schofield@live.com>
> wrote:
>
> > Hi Jun,
> > Thanks for the questions.
> >
> > 1. The PartitionMetadata is only stored if there is rack ID information.
> > So, there are situations in which it’s redundant, but almost always the
> > PartitionMetadata will be empty.
> >
> > 2. They are for subtly different purposes. ShareGroupPartitionMetadata
> > is essentially a persistent cache of the metadata for topic-partitions
> > which
> > can be assigned to group members. ShareGroupStatePartitionMetadata
> > is keeping track of which partitions have share-group state in the share
> > coordinator. I prefer having them separate, in particular because it
> means
> > the ShareGroupPartitionMetadata matches nicely with consumer groups,
> > and also because share groups are able to work without a share
> coordinator
> > (the persister is pluggable).
> >
> > I propose that as we develop the code in the group coordinator which uses
> > the various ShareGroupState RPCs to talk to the share coordinator,
> > we will confirm whether the record schemas are optimal
> > and make a further refinement if required. Let me know if that’s OK.
> >
> > Thanks,
> > Andrew
> >
> >
> > > On 14 Aug 2024, at 00:00, Jun Rao <ju...@confluent.io.INVALID> wrote:
> > >
> > > Hi, Andrew,
> > >
> > > Thanks for the update. A couple of followup questions.
> > >
> > > 1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given
> > > PartitionMetadata?
> > >
> > > 2. I assume that ShareGroupPartitionMetadataValue contains initialized
> > > partitions. Does that
> > > make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant?
> > >
> > > Jun
> > >
> > > On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield <
> > andrew_schofield@live.com>
> > > wrote:
> > >
> > >> Hi,
> > >> During the implementation of the group management code for KIP-932,
> > >> we learnt that the decision not to persist assignments for share
> groups
> > was
> > >> making it difficult to fit with the coordinator framework, in
> particular
> > >> the
> > >> use of timeline structures. As a result, I decided to change the KIP
> to
> > >> follow the consumer group record scheme much more closely.
> > >>
> > >> Now, share groups do persist assignments and member epochs which
> > >> resolves the problems we were experiencing managing soft state in the
> > >> coordinator and making group membership reliable across coordinator
> > >> restarts.
> > >>
> > >> I have updated the KIP accordingly and the revised code has been
> merged.
> > >> If anyone has any comments, please let me know.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>
> > >>> On 21 May 2024, at 15:35, Andrew Schofield <
> andrew_schofield@live.com>
> > >> wrote:
> > >>>
> > >>> Hi Jun,
> > >>> All the client metrics are standard. None are required.
> > >>>
> > >>> I’ve updated the KIP accordingly.
> > >>>
> > >>> Thanks,
> > >>> Andrew
> > >>>
> > >>>> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
> > >>>>
> > >>>> Hi, Andrew,
> > >>>>
> > >>>> Thanks for the update. Should we mark whether those metrics are
> > >>>> standard/required for KIP-714?
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> > >> andrew_schofield@live.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>> I have made a small update to the KIP as a result of testing the
> new
> > >>>>> share consumer with client telemetry (KIP-714).
> > >>>>>
> > >>>>> I’ve added telemetry metric names to the table of client metrics
> and
> > >>>>> also updated the metric group names so that the resulting client
> > >> metrics
> > >>>>> sent to the broker have consistent names.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Andrew
> > >>>>>
> > >>>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com>
> > wrote:
> > >>>>>>
> > >>>>>> Hi Andrew,
> > >>>>>>
> > >>>>>> Thanks for the KIP. Great write-up!
> > >>>>>>
> > >>>>>> +1 (binding)
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> > >> satish.duggana@gmail.com>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Andrew,
> > >>>>>>> Thanks for the nice KIP, it will allow other messaging use cases
> to
> > >> be
> > >>>>>>> onboarded to Kafka.
> > >>>>>>>
> > >>>>>>> +1 from me.
> > >>>>>>>
> > >>>>>>> Satish.
> > >>>>>>>
> > >>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
> > >> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi, Andrew,
> > >>>>>>>>
> > >>>>>>>> Thanks for the KIP. +1
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> > >> edoardlists@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks Andrew,
> > >>>>>>>>>
> > >>>>>>>>> +1 (binding)
> > >>>>>>>>>
> > >>>>>>>>> Edo
> > >>>>>>>>>
> > >>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> > >>>>>>>>> <ke...@cloudflare.com.invalid> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hi Andrew
> > >>>>>>>>>>
> > >>>>>>>>>> + 1 (Non-Binding)
> > >>>>>>>>>>
> > >>>>>>>>>> This will be great addition to Kafka
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> > >>>>> apoorvmittal10@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Andrew,
> > >>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a
> > valuable
> > >>>>>>>>> addition
> > >>>>>>>>>>> to the Kafka, excited to see the KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> + 1 (Non-Binding)
> > >>>>>>>>>>>
> > >>>>>>>>>>> Regards,
> > >>>>>>>>>>> Apoorv Mittal
> > >>>>>>>>>>> +44 7721681581
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > >>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few
> months
> > >> and
> > >>>>>>>>>>>> discussions have quietened down.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I’d like to open the voting for KIP-932:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Andrew
> >
> >
> >
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Jun Rao <ju...@confluent.io.INVALID>.
Hi, Andrew,
Thanks for the updates. Looks reasonable to me.
I don't see ShareConsumer.acknowledge(String, int, long, AcknowledgeType)
in the KIP or the code. There are only the following two.
void acknowledge(ConsumerRecord<K, V> record);
void acknowledge(ConsumerRecord<K, V> record, AcknowledgeType type);
Thanks,
Jun
On Fri, Jul 11, 2025 at 2:30 AM Andrew Schofield <
andrew_schofield_jira@outlook.com> wrote:
> Hi,
> As we've written the code for KIP-932, we've made some small changes to
> the KIP
> to correct mistakes, align with other accepted KIPs and fix some omissions
> picked
> up during PR reviews. We also have experience of people writing
> applications
> against the ShareConsumer interface and have corrected a couple of
> problems.
>
> * Using share.version=1 as the switch to enable share groups. This aligns
> with group.version=1 for KIP-848 and streams.version=1 for KIP-1071.
> * Aligned with KIP-1043 by using a common GroupState and removing
> Admin.listShareGroups (because Admin.listGroups handles all group types).
> * Corrected the topic authorizations on admin requests.
> * Added SHARE_SESSION_LIMIT_REACHED error code to give proper
> consumer error handling when the limit is reached.
> * Adjusted ShareFetch RPC request to remove PartitionMaxBytes and
> add BatchSize to improve efficiency of fetching and state management.
> * Changed the rules for explicit acknowledgement because, as originally
> written, it was simple to write an application that went into an infinite
> loop.
> Now, when using explicit acknowledgement, it is mandatory to acknowledge
> every single record, which allows an exception to be thrown if the
> application
> does not do this properly.
> * Added a method overload with signature
> ShareConsumer.acknowledge(String, int, long, AcknowledgeType) to go with
> ShareConsumer.acknowledge(ConsumerRecord, AcknowledgeType).
> This is because the KIP said it would be possible to choose the acknowledge
> type in the event of RecordDeserializationException. However, because the
> application does not have the ConsumerRecord instance when the
> deserialization
> fails, but it does have topic name, partition and offset, adding an
> overloaded
> method enables the application to choose the acknowledge type.
> * The SimpleAssignor no longer uses hashing as part of making assignments.
> In practice, this hindered the assignment computation. SimpleAssignor
> achieves
> the balance and stickiness described previously, just without hashing.
> * Added
> org.apache.kafka.coordinator.group.api.assignor.GroupSpec.isPartitionAssignable
> so that partitions which are in the process of being initialized in the
> share coordinator
> can be omitted from assignment computation.
> * Removed the ShareGroupPartitionMetadata record type (KIP-1101).
> * Added timestamps to ShareSnapshot records so it is possible to manage
> pruning of records correctly.
> * Adjusted some of the config defaults and limits such as maximum for lock
> duration which didn't make sense.
>
> KIP-932 mentions some future work and I have started to create new KIPs
> for this.
>
> Thanks,
> Andrew
> ________________________________________
> From: Jun Rao <ju...@confluent.io.INVALID>
> Sent: 15 August 2024 19:07
> To: dev@kafka.apache.org <de...@kafka.apache.org>
> Subject: Re: [VOTE] KIP-932: Queues for Kafka
>
> Hi, Andrew,
>
> Thanks for the clarification. Sounds good.
>
> Jun
>
> On Thu, Aug 15, 2024 at 2:53 AM Andrew Schofield <
> andrew_schofield@live.com>
> wrote:
>
> > Hi Jun,
> > Thanks for the questions.
> >
> > 1. The PartitionMetadata is only stored if there is rack ID information.
> > So, there are situations in which it’s redundant, but almost always the
> > PartitionMetadata will be empty.
> >
> > 2. They are for subtly different purposes. ShareGroupPartitionMetadata
> > is essentially a persistent cache of the metadata for topic-partitions
> > which
> > can be assigned to group members. ShareGroupStatePartitionMetadata
> > is keeping track of which partitions have share-group state in the share
> > coordinator. I prefer having them separate, in particular because it
> means
> > the ShareGroupPartitionMetadata matches nicely with consumer groups,
> > and also because share groups are able to work without a share
> coordinator
> > (the persister is pluggable).
> >
> > I propose that as we develop the code in the group coordinator which uses
> > the various ShareGroupState RPCs to talk to the share coordinator,
> > we will confirm whether the record schemas are optimal
> > and make a further refinement if required. Let me know if that’s OK.
> >
> > Thanks,
> > Andrew
> >
> >
> > > On 14 Aug 2024, at 00:00, Jun Rao <ju...@confluent.io.INVALID> wrote:
> > >
> > > Hi, Andrew,
> > >
> > > Thanks for the update. A couple of followup questions.
> > >
> > > 1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given
> > > PartitionMetadata?
> > >
> > > 2. I assume that ShareGroupPartitionMetadataValue contains initialized
> > > partitions. Does that
> > > make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant?
> > >
> > > Jun
> > >
> > > On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield <
> > andrew_schofield@live.com>
> > > wrote:
> > >
> > >> Hi,
> > >> During the implementation of the group management code for KIP-932,
> > >> we learnt that the decision not to persist assignments for share
> groups
> > was
> > >> making it difficult to fit with the coordinator framework, in
> particular
> > >> the
> > >> use of timeline structures. As a result, I decided to change the KIP
> to
> > >> follow the consumer group record scheme much more closely.
> > >>
> > >> Now, share groups do persist assignments and member epochs which
> > >> resolves the problems we were experiencing managing soft state in the
> > >> coordinator and making group membership reliable across coordinator
> > >> restarts.
> > >>
> > >> I have updated the KIP accordingly and the revised code has been
> merged.
> > >> If anyone has any comments, please let me know.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>
> > >>> On 21 May 2024, at 15:35, Andrew Schofield <
> andrew_schofield@live.com>
> > >> wrote:
> > >>>
> > >>> Hi Jun,
> > >>> All the client metrics are standard. None are required.
> > >>>
> > >>> I’ve updated the KIP accordingly.
> > >>>
> > >>> Thanks,
> > >>> Andrew
> > >>>
> > >>>> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
> > >>>>
> > >>>> Hi, Andrew,
> > >>>>
> > >>>> Thanks for the update. Should we mark whether those metrics are
> > >>>> standard/required for KIP-714?
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> > >> andrew_schofield@live.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>> I have made a small update to the KIP as a result of testing the
> new
> > >>>>> share consumer with client telemetry (KIP-714).
> > >>>>>
> > >>>>> I’ve added telemetry metric names to the table of client metrics
> and
> > >>>>> also updated the metric group names so that the resulting client
> > >> metrics
> > >>>>> sent to the broker have consistent names.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Andrew
> > >>>>>
> > >>>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com>
> > wrote:
> > >>>>>>
> > >>>>>> Hi Andrew,
> > >>>>>>
> > >>>>>> Thanks for the KIP. Great write-up!
> > >>>>>>
> > >>>>>> +1 (binding)
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> > >> satish.duggana@gmail.com>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Andrew,
> > >>>>>>> Thanks for the nice KIP, it will allow other messaging use cases
> to
> > >> be
> > >>>>>>> onboarded to Kafka.
> > >>>>>>>
> > >>>>>>> +1 from me.
> > >>>>>>>
> > >>>>>>> Satish.
> > >>>>>>>
> > >>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
> > >> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi, Andrew,
> > >>>>>>>>
> > >>>>>>>> Thanks for the KIP. +1
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> > >> edoardlists@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks Andrew,
> > >>>>>>>>>
> > >>>>>>>>> +1 (binding)
> > >>>>>>>>>
> > >>>>>>>>> Edo
> > >>>>>>>>>
> > >>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> > >>>>>>>>> <ke...@cloudflare.com.invalid> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hi Andrew
> > >>>>>>>>>>
> > >>>>>>>>>> + 1 (Non-Binding)
> > >>>>>>>>>>
> > >>>>>>>>>> This will be great addition to Kafka
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> > >>>>> apoorvmittal10@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Andrew,
> > >>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a
> > valuable
> > >>>>>>>>> addition
> > >>>>>>>>>>> to the Kafka, excited to see the KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> + 1 (Non-Binding)
> > >>>>>>>>>>>
> > >>>>>>>>>>> Regards,
> > >>>>>>>>>>> Apoorv Mittal
> > >>>>>>>>>>> +44 7721681581
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > >>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few
> months
> > >> and
> > >>>>>>>>>>>> discussions have quietened down.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I’d like to open the voting for KIP-932:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Andrew
> >
> >
> >
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Andrew Schofield <an...@outlook.com>.
Hi,
As we've written the code for KIP-932, we've made some small changes to the KIP
to correct mistakes, align with other accepted KIPs and fix some omissions picked
up during PR reviews. We also have experience of people writing applications
against the ShareConsumer interface and have corrected a couple of problems.
* Using share.version=1 as the switch to enable share groups. This aligns
with group.version=1 for KIP-848 and streams.version=1 for KIP-1071.
* Aligned with KIP-1043 by using a common GroupState and removing
Admin.listShareGroups (because Admin.listGroups handles all group types).
* Corrected the topic authorizations on admin requests.
* Added SHARE_SESSION_LIMIT_REACHED error code to give proper
consumer error handling when the limit is reached.
* Adjusted ShareFetch RPC request to remove PartitionMaxBytes and
add BatchSize to improve efficiency of fetching and state management.
* Changed the rules for explicit acknowledgement because, as originally
written, it was simple to write an application that went into an infinite loop.
Now, when using explicit acknowledgement, it is mandatory to acknowledge
every single record, which allows an exception to be thrown if the application
does not do this properly.
* Added a method overload with signature
ShareConsumer.acknowledge(String, int, long, AcknowledgeType) to go with
ShareConsumer.acknowledge(ConsumerRecord, AcknowledgeType).
This is because the KIP said it would be possible to choose the acknowledge
type in the event of RecordDeserializationException. However, because the
application does not have the ConsumerRecord instance when the deserialization
fails, but it does have topic name, partition and offset, adding an overloaded
method enables the application to choose the acknowledge type.
* The SimpleAssignor no longer uses hashing as part of making assignments.
In practice, this hindered the assignment computation. SimpleAssignor achieves
the balance and stickiness described previously, just without hashing.
* Added org.apache.kafka.coordinator.group.api.assignor.GroupSpec.isPartitionAssignable
so that partitions which are in the process of being initialized in the share coordinator
can be omitted from assignment computation.
* Removed the ShareGroupPartitionMetadata record type (KIP-1101).
* Added timestamps to ShareSnapshot records so it is possible to manage
pruning of records correctly.
* Adjusted some of the config defaults and limits such as maximum for lock
duration which didn't make sense.
KIP-932 mentions some future work and I have started to create new KIPs for this.
Thanks,
Andrew
________________________________________
From: Jun Rao <ju...@confluent.io.INVALID>
Sent: 15 August 2024 19:07
To: dev@kafka.apache.org <de...@kafka.apache.org>
Subject: Re: [VOTE] KIP-932: Queues for Kafka
Hi, Andrew,
Thanks for the clarification. Sounds good.
Jun
On Thu, Aug 15, 2024 at 2:53 AM Andrew Schofield <an...@live.com>
wrote:
> Hi Jun,
> Thanks for the questions.
>
> 1. The PartitionMetadata is only stored if there is rack ID information.
> So, there are situations in which it’s redundant, but almost always the
> PartitionMetadata will be empty.
>
> 2. They are for subtly different purposes. ShareGroupPartitionMetadata
> is essentially a persistent cache of the metadata for topic-partitions
> which
> can be assigned to group members. ShareGroupStatePartitionMetadata
> is keeping track of which partitions have share-group state in the share
> coordinator. I prefer having them separate, in particular because it means
> the ShareGroupPartitionMetadata matches nicely with consumer groups,
> and also because share groups are able to work without a share coordinator
> (the persister is pluggable).
>
> I propose that as we develop the code in the group coordinator which uses
> the various ShareGroupState RPCs to talk to the share coordinator,
> we will confirm whether the record schemas are optimal
> and make a further refinement if required. Let me know if that’s OK.
>
> Thanks,
> Andrew
>
>
> > On 14 Aug 2024, at 00:00, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >
> > Hi, Andrew,
> >
> > Thanks for the update. A couple of followup questions.
> >
> > 1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given
> > PartitionMetadata?
> >
> > 2. I assume that ShareGroupPartitionMetadataValue contains initialized
> > partitions. Does that
> > make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant?
> >
> > Jun
> >
> > On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield <
> andrew_schofield@live.com>
> > wrote:
> >
> >> Hi,
> >> During the implementation of the group management code for KIP-932,
> >> we learnt that the decision not to persist assignments for share groups
> was
> >> making it difficult to fit with the coordinator framework, in particular
> >> the
> >> use of timeline structures. As a result, I decided to change the KIP to
> >> follow the consumer group record scheme much more closely.
> >>
> >> Now, share groups do persist assignments and member epochs which
> >> resolves the problems we were experiencing managing soft state in the
> >> coordinator and making group membership reliable across coordinator
> >> restarts.
> >>
> >> I have updated the KIP accordingly and the revised code has been merged.
> >> If anyone has any comments, please let me know.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>
> >>> On 21 May 2024, at 15:35, Andrew Schofield <an...@live.com>
> >> wrote:
> >>>
> >>> Hi Jun,
> >>> All the client metrics are standard. None are required.
> >>>
> >>> I’ve updated the KIP accordingly.
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>>> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >>>>
> >>>> Hi, Andrew,
> >>>>
> >>>> Thanks for the update. Should we mark whether those metrics are
> >>>> standard/required for KIP-714?
> >>>>
> >>>> Jun
> >>>>
> >>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> >> andrew_schofield@live.com>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>> I have made a small update to the KIP as a result of testing the new
> >>>>> share consumer with client telemetry (KIP-714).
> >>>>>
> >>>>> I’ve added telemetry metric names to the table of client metrics and
> >>>>> also updated the metric group names so that the resulting client
> >> metrics
> >>>>> sent to the broker have consistent names.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Hi Andrew,
> >>>>>>
> >>>>>> Thanks for the KIP. Great write-up!
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> >> satish.duggana@gmail.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>> Hi Andrew,
> >>>>>>> Thanks for the nice KIP, it will allow other messaging use cases to
> >> be
> >>>>>>> onboarded to Kafka.
> >>>>>>>
> >>>>>>> +1 from me.
> >>>>>>>
> >>>>>>> Satish.
> >>>>>>>
> >>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
> >> wrote:
> >>>>>>>>
> >>>>>>>> Hi, Andrew,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP. +1
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> >> edoardlists@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks Andrew,
> >>>>>>>>>
> >>>>>>>>> +1 (binding)
> >>>>>>>>>
> >>>>>>>>> Edo
> >>>>>>>>>
> >>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> >>>>>>>>> <ke...@cloudflare.com.invalid> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Andrew
> >>>>>>>>>>
> >>>>>>>>>> + 1 (Non-Binding)
> >>>>>>>>>>
> >>>>>>>>>> This will be great addition to Kafka
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> >>>>> apoorvmittal10@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Andrew,
> >>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a
> valuable
> >>>>>>>>> addition
> >>>>>>>>>>> to the Kafka, excited to see the KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> + 1 (Non-Binding)
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Apoorv Mittal
> >>>>>>>>>>> +44 7721681581
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> >>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few months
> >> and
> >>>>>>>>>>>> discussions have quietened down.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I’d like to open the voting for KIP-932:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Andrew
>
>
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Jun Rao <ju...@confluent.io.INVALID>.
Hi, Andrew,
Thanks for the clarification. Sounds good.
Jun
On Thu, Aug 15, 2024 at 2:53 AM Andrew Schofield <an...@live.com>
wrote:
> Hi Jun,
> Thanks for the questions.
>
> 1. The PartitionMetadata is only stored if there is rack ID information.
> So, there are situations in which it’s redundant, but almost always the
> PartitionMetadata will be empty.
>
> 2. They are for subtly different purposes. ShareGroupPartitionMetadata
> is essentially a persistent cache of the metadata for topic-partitions
> which
> can be assigned to group members. ShareGroupStatePartitionMetadata
> is keeping track of which partitions have share-group state in the share
> coordinator. I prefer having them separate, in particular because it means
> the ShareGroupPartitionMetadata matches nicely with consumer groups,
> and also because share groups are able to work without a share coordinator
> (the persister is pluggable).
>
> I propose that as we develop the code in the group coordinator which uses
> the various ShareGroupState RPCs to talk to the share coordinator,
> we will confirm whether the record schemas are optimal
> and make a further refinement if required. Let me know if that’s OK.
>
> Thanks,
> Andrew
>
>
> > On 14 Aug 2024, at 00:00, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >
> > Hi, Andrew,
> >
> > Thanks for the update. A couple of followup questions.
> >
> > 1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given
> > PartitionMetadata?
> >
> > 2. I assume that ShareGroupPartitionMetadataValue contains initialized
> > partitions. Does that
> > make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant?
> >
> > Jun
> >
> > On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield <
> andrew_schofield@live.com>
> > wrote:
> >
> >> Hi,
> >> During the implementation of the group management code for KIP-932,
> >> we learnt that the decision not to persist assignments for share groups
> was
> >> making it difficult to fit with the coordinator framework, in particular
> >> the
> >> use of timeline structures. As a result, I decided to change the KIP to
> >> follow the consumer group record scheme much more closely.
> >>
> >> Now, share groups do persist assignments and member epochs which
> >> resolves the problems we were experiencing managing soft state in the
> >> coordinator and making group membership reliable across coordinator
> >> restarts.
> >>
> >> I have updated the KIP accordingly and the revised code has been merged.
> >> If anyone has any comments, please let me know.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>
> >>> On 21 May 2024, at 15:35, Andrew Schofield <an...@live.com>
> >> wrote:
> >>>
> >>> Hi Jun,
> >>> All the client metrics are standard. None are required.
> >>>
> >>> I’ve updated the KIP accordingly.
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>>> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >>>>
> >>>> Hi, Andrew,
> >>>>
> >>>> Thanks for the update. Should we mark whether those metrics are
> >>>> standard/required for KIP-714?
> >>>>
> >>>> Jun
> >>>>
> >>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> >> andrew_schofield@live.com>
> >>>> wrote:
> >>>>
> >>>>> Hi,
> >>>>> I have made a small update to the KIP as a result of testing the new
> >>>>> share consumer with client telemetry (KIP-714).
> >>>>>
> >>>>> I’ve added telemetry metric names to the table of client metrics and
> >>>>> also updated the metric group names so that the resulting client
> >> metrics
> >>>>> sent to the broker have consistent names.
> >>>>>
> >>>>> Thanks,
> >>>>> Andrew
> >>>>>
> >>>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Hi Andrew,
> >>>>>>
> >>>>>> Thanks for the KIP. Great write-up!
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> >> satish.duggana@gmail.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>> Hi Andrew,
> >>>>>>> Thanks for the nice KIP, it will allow other messaging use cases to
> >> be
> >>>>>>> onboarded to Kafka.
> >>>>>>>
> >>>>>>> +1 from me.
> >>>>>>>
> >>>>>>> Satish.
> >>>>>>>
> >>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
> >> wrote:
> >>>>>>>>
> >>>>>>>> Hi, Andrew,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP. +1
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> >> edoardlists@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks Andrew,
> >>>>>>>>>
> >>>>>>>>> +1 (binding)
> >>>>>>>>>
> >>>>>>>>> Edo
> >>>>>>>>>
> >>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> >>>>>>>>> <ke...@cloudflare.com.invalid> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Andrew
> >>>>>>>>>>
> >>>>>>>>>> + 1 (Non-Binding)
> >>>>>>>>>>
> >>>>>>>>>> This will be great addition to Kafka
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> >>>>> apoorvmittal10@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Andrew,
> >>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a
> valuable
> >>>>>>>>> addition
> >>>>>>>>>>> to the Kafka, excited to see the KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> + 1 (Non-Binding)
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Apoorv Mittal
> >>>>>>>>>>> +44 7721681581
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> >>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few months
> >> and
> >>>>>>>>>>>> discussions have quietened down.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I’d like to open the voting for KIP-932:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Andrew
>
>
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Andrew Schofield <an...@live.com>.
Hi Jun,
Thanks for the questions.
1. The PartitionMetadata is only stored if there is rack ID information.
So, there are situations in which it’s redundant, but almost always the
PartitionMetadata will be empty.
2. They are for subtly different purposes. ShareGroupPartitionMetadata
is essentially a persistent cache of the metadata for topic-partitions which
can be assigned to group members. ShareGroupStatePartitionMetadata
is keeping track of which partitions have share-group state in the share
coordinator. I prefer having them separate, in particular because it means
the ShareGroupPartitionMetadata matches nicely with consumer groups,
and also because share groups are able to work without a share coordinator
(the persister is pluggable).
I propose that as we develop the code in the group coordinator which uses
the various ShareGroupState RPCs to talk to the share coordinator,
we will confirm whether the record schemas are optimal
and make a further refinement if required. Let me know if that’s OK.
Thanks,
Andrew
> On 14 Aug 2024, at 00:00, Jun Rao <ju...@confluent.io.INVALID> wrote:
>
> Hi, Andrew,
>
> Thanks for the update. A couple of followup questions.
>
> 1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given
> PartitionMetadata?
>
> 2. I assume that ShareGroupPartitionMetadataValue contains initialized
> partitions. Does that
> make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant?
>
> Jun
>
> On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield <an...@live.com>
> wrote:
>
>> Hi,
>> During the implementation of the group management code for KIP-932,
>> we learnt that the decision not to persist assignments for share groups was
>> making it difficult to fit with the coordinator framework, in particular
>> the
>> use of timeline structures. As a result, I decided to change the KIP to
>> follow the consumer group record scheme much more closely.
>>
>> Now, share groups do persist assignments and member epochs which
>> resolves the problems we were experiencing managing soft state in the
>> coordinator and making group membership reliable across coordinator
>> restarts.
>>
>> I have updated the KIP accordingly and the revised code has been merged.
>> If anyone has any comments, please let me know.
>>
>> Thanks,
>> Andrew
>>
>>
>>> On 21 May 2024, at 15:35, Andrew Schofield <an...@live.com>
>> wrote:
>>>
>>> Hi Jun,
>>> All the client metrics are standard. None are required.
>>>
>>> I’ve updated the KIP accordingly.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
>>>>
>>>> Hi, Andrew,
>>>>
>>>> Thanks for the update. Should we mark whether those metrics are
>>>> standard/required for KIP-714?
>>>>
>>>> Jun
>>>>
>>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
>> andrew_schofield@live.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> I have made a small update to the KIP as a result of testing the new
>>>>> share consumer with client telemetry (KIP-714).
>>>>>
>>>>> I’ve added telemetry metric names to the table of client metrics and
>>>>> also updated the metric group names so that the resulting client
>> metrics
>>>>> sent to the broker have consistent names.
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> Thanks for the KIP. Great write-up!
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
>> satish.duggana@gmail.com>
>>>>> wrote:
>>>>>>>
>>>>>>> Hi Andrew,
>>>>>>> Thanks for the nice KIP, it will allow other messaging use cases to
>> be
>>>>>>> onboarded to Kafka.
>>>>>>>
>>>>>>> +1 from me.
>>>>>>>
>>>>>>> Satish.
>>>>>>>
>>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
>> wrote:
>>>>>>>>
>>>>>>>> Hi, Andrew,
>>>>>>>>
>>>>>>>> Thanks for the KIP. +1
>>>>>>>>
>>>>>>>> Jun
>>>>>>>>
>>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
>> edoardlists@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Andrew,
>>>>>>>>>
>>>>>>>>> +1 (binding)
>>>>>>>>>
>>>>>>>>> Edo
>>>>>>>>>
>>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
>>>>>>>>> <ke...@cloudflare.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Andrew
>>>>>>>>>>
>>>>>>>>>> + 1 (Non-Binding)
>>>>>>>>>>
>>>>>>>>>> This will be great addition to Kafka
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
>>>>> apoorvmittal10@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
>>>>>>>>> addition
>>>>>>>>>>> to the Kafka, excited to see the KIP.
>>>>>>>>>>>
>>>>>>>>>>> + 1 (Non-Binding)
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Apoorv Mittal
>>>>>>>>>>> +44 7721681581
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
>>>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few months
>> and
>>>>>>>>>>>> discussions have quietened down.
>>>>>>>>>>>>
>>>>>>>>>>>> I’d like to open the voting for KIP-932:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Andrew
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Jun Rao <ju...@confluent.io.INVALID>.
Hi, Andrew,
Thanks for the update. A couple of followup questions.
1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given
PartitionMetadata?
2. I assume that ShareGroupPartitionMetadataValue contains initialized
partitions. Does that
make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant?
Jun
On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield <an...@live.com>
wrote:
> Hi,
> During the implementation of the group management code for KIP-932,
> we learnt that the decision not to persist assignments for share groups was
> making it difficult to fit with the coordinator framework, in particular
> the
> use of timeline structures. As a result, I decided to change the KIP to
> follow the consumer group record scheme much more closely.
>
> Now, share groups do persist assignments and member epochs which
> resolves the problems we were experiencing managing soft state in the
> coordinator and making group membership reliable across coordinator
> restarts.
>
> I have updated the KIP accordingly and the revised code has been merged.
> If anyone has any comments, please let me know.
>
> Thanks,
> Andrew
>
>
> > On 21 May 2024, at 15:35, Andrew Schofield <an...@live.com>
> wrote:
> >
> > Hi Jun,
> > All the client metrics are standard. None are required.
> >
> > I’ve updated the KIP accordingly.
> >
> > Thanks,
> > Andrew
> >
> >> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
> >>
> >> Hi, Andrew,
> >>
> >> Thanks for the update. Should we mark whether those metrics are
> >> standard/required for KIP-714?
> >>
> >> Jun
> >>
> >> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <
> andrew_schofield@live.com>
> >> wrote:
> >>
> >>> Hi,
> >>> I have made a small update to the KIP as a result of testing the new
> >>> share consumer with client telemetry (KIP-714).
> >>>
> >>> I’ve added telemetry metric names to the table of client metrics and
> >>> also updated the metric group names so that the resulting client
> metrics
> >>> sent to the broker have consistent names.
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
> >>>>
> >>>> Hi Andrew,
> >>>>
> >>>> Thanks for the KIP. Great write-up!
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>> Thanks,
> >>>>
> >>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <
> satish.duggana@gmail.com>
> >>> wrote:
> >>>>>
> >>>>> Hi Andrew,
> >>>>> Thanks for the nice KIP, it will allow other messaging use cases to
> be
> >>>>> onboarded to Kafka.
> >>>>>
> >>>>> +1 from me.
> >>>>>
> >>>>> Satish.
> >>>>>
> >>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >>>>>>
> >>>>>> Hi, Andrew,
> >>>>>>
> >>>>>> Thanks for the KIP. +1
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <
> edoardlists@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Andrew,
> >>>>>>>
> >>>>>>> +1 (binding)
> >>>>>>>
> >>>>>>> Edo
> >>>>>>>
> >>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> >>>>>>> <ke...@cloudflare.com.invalid> wrote:
> >>>>>>>>
> >>>>>>>> Hi Andrew
> >>>>>>>>
> >>>>>>>> + 1 (Non-Binding)
> >>>>>>>>
> >>>>>>>> This will be great addition to Kafka
> >>>>>>>>
> >>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> >>> apoorvmittal10@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Andrew,
> >>>>>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
> >>>>>>> addition
> >>>>>>>>> to the Kafka, excited to see the KIP.
> >>>>>>>>>
> >>>>>>>>> + 1 (Non-Binding)
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Apoorv Mittal
> >>>>>>>>> +44 7721681581
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> >>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>> I’ve been working to complete KIP-932 over the past few months
> and
> >>>>>>>>>> discussions have quietened down.
> >>>>>>>>>>
> >>>>>>>>>> I’d like to open the voting for KIP-932:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Andrew
>
>
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Andrew Schofield <an...@live.com>.
Hi,
During the implementation of the group management code for KIP-932,
we learnt that the decision not to persist assignments for share groups was
making it difficult to fit with the coordinator framework, in particular the
use of timeline structures. As a result, I decided to change the KIP to
follow the consumer group record scheme much more closely.
Now, share groups do persist assignments and member epochs which
resolves the problems we were experiencing managing soft state in the
coordinator and making group membership reliable across coordinator
restarts.
I have updated the KIP accordingly and the revised code has been merged.
If anyone has any comments, please let me know.
Thanks,
Andrew
> On 21 May 2024, at 15:35, Andrew Schofield <an...@live.com> wrote:
>
> Hi Jun,
> All the client metrics are standard. None are required.
>
> I’ve updated the KIP accordingly.
>
> Thanks,
> Andrew
>
>> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
>>
>> Hi, Andrew,
>>
>> Thanks for the update. Should we mark whether those metrics are
>> standard/required for KIP-714?
>>
>> Jun
>>
>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <an...@live.com>
>> wrote:
>>
>>> Hi,
>>> I have made a small update to the KIP as a result of testing the new
>>> share consumer with client telemetry (KIP-714).
>>>
>>> I’ve added telemetry metric names to the table of client metrics and
>>> also updated the metric group names so that the resulting client metrics
>>> sent to the broker have consistent names.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
>>>>
>>>> Hi Andrew,
>>>>
>>>> Thanks for the KIP. Great write-up!
>>>>
>>>> +1 (binding)
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <sa...@gmail.com>
>>> wrote:
>>>>>
>>>>> Hi Andrew,
>>>>> Thanks for the nice KIP, it will allow other messaging use cases to be
>>>>> onboarded to Kafka.
>>>>>
>>>>> +1 from me.
>>>>>
>>>>> Satish.
>>>>>
>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid> wrote:
>>>>>>
>>>>>> Hi, Andrew,
>>>>>>
>>>>>> Thanks for the KIP. +1
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <ed...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Andrew,
>>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> Edo
>>>>>>>
>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
>>>>>>> <ke...@cloudflare.com.invalid> wrote:
>>>>>>>>
>>>>>>>> Hi Andrew
>>>>>>>>
>>>>>>>> + 1 (Non-Binding)
>>>>>>>>
>>>>>>>> This will be great addition to Kafka
>>>>>>>>
>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
>>> apoorvmittal10@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Andrew,
>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
>>>>>>> addition
>>>>>>>>> to the Kafka, excited to see the KIP.
>>>>>>>>>
>>>>>>>>> + 1 (Non-Binding)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Apoorv Mittal
>>>>>>>>> +44 7721681581
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
>>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I’ve been working to complete KIP-932 over the past few months and
>>>>>>>>>> discussions have quietened down.
>>>>>>>>>>
>>>>>>>>>> I’d like to open the voting for KIP-932:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Andrew
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Andrew Schofield <an...@live.com>.
Hi Jun,
All the client metrics are standard. None are required.
I’ve updated the KIP accordingly.
Thanks,
Andrew
> On 15 May 2024, at 21:36, Jun Rao <ju...@confluent.io.INVALID> wrote:
>
> Hi, Andrew,
>
> Thanks for the update. Should we mark whether those metrics are
> standard/required for KIP-714?
>
> Jun
>
> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <an...@live.com>
> wrote:
>
>> Hi,
>> I have made a small update to the KIP as a result of testing the new
>> share consumer with client telemetry (KIP-714).
>>
>> I’ve added telemetry metric names to the table of client metrics and
>> also updated the metric group names so that the resulting client metrics
>> sent to the broker have consistent names.
>>
>> Thanks,
>> Andrew
>>
>>> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
>>>
>>> Hi Andrew,
>>>
>>> Thanks for the KIP. Great write-up!
>>>
>>> +1 (binding)
>>>
>>> Thanks,
>>>
>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <sa...@gmail.com>
>> wrote:
>>>>
>>>> Hi Andrew,
>>>> Thanks for the nice KIP, it will allow other messaging use cases to be
>>>> onboarded to Kafka.
>>>>
>>>> +1 from me.
>>>>
>>>> Satish.
>>>>
>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid> wrote:
>>>>>
>>>>> Hi, Andrew,
>>>>>
>>>>> Thanks for the KIP. +1
>>>>>
>>>>> Jun
>>>>>
>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <ed...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Andrew,
>>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> Edo
>>>>>>
>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
>>>>>> <ke...@cloudflare.com.invalid> wrote:
>>>>>>>
>>>>>>> Hi Andrew
>>>>>>>
>>>>>>> + 1 (Non-Binding)
>>>>>>>
>>>>>>> This will be great addition to Kafka
>>>>>>>
>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
>> apoorvmittal10@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
>>>>>> addition
>>>>>>>> to the Kafka, excited to see the KIP.
>>>>>>>>
>>>>>>>> + 1 (Non-Binding)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Apoorv Mittal
>>>>>>>> +44 7721681581
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
>>>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I’ve been working to complete KIP-932 over the past few months and
>>>>>>>>> discussions have quietened down.
>>>>>>>>>
>>>>>>>>> I’d like to open the voting for KIP-932:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Andrew
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Jun Rao <ju...@confluent.io.INVALID>.
Hi, Andrew,
Thanks for the update. Should we mark whether those metrics are
standard/required for KIP-714?
Jun
On Tue, May 14, 2024 at 7:31 AM Andrew Schofield <an...@live.com>
wrote:
> Hi,
> I have made a small update to the KIP as a result of testing the new
> share consumer with client telemetry (KIP-714).
>
> I’ve added telemetry metric names to the table of client metrics and
> also updated the metric group names so that the resulting client metrics
> sent to the broker have consistent names.
>
> Thanks,
> Andrew
>
> > On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
> >
> > Hi Andrew,
> >
> > Thanks for the KIP. Great write-up!
> >
> > +1 (binding)
> >
> > Thanks,
> >
> > On Wed, May 8, 2024 at 12:17 PM Satish Duggana <sa...@gmail.com>
> wrote:
> >>
> >> Hi Andrew,
> >> Thanks for the nice KIP, it will allow other messaging use cases to be
> >> onboarded to Kafka.
> >>
> >> +1 from me.
> >>
> >> Satish.
> >>
> >> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid> wrote:
> >>>
> >>> Hi, Andrew,
> >>>
> >>> Thanks for the KIP. +1
> >>>
> >>> Jun
> >>>
> >>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <ed...@gmail.com>
> >>> wrote:
> >>>
> >>>> Thanks Andrew,
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>> Edo
> >>>>
> >>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> >>>> <ke...@cloudflare.com.invalid> wrote:
> >>>>>
> >>>>> Hi Andrew
> >>>>>
> >>>>> + 1 (Non-Binding)
> >>>>>
> >>>>> This will be great addition to Kafka
> >>>>>
> >>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <
> apoorvmittal10@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Andrew,
> >>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
> >>>> addition
> >>>>>> to the Kafka, excited to see the KIP.
> >>>>>>
> >>>>>> + 1 (Non-Binding)
> >>>>>>
> >>>>>> Regards,
> >>>>>> Apoorv Mittal
> >>>>>> +44 7721681581
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> >>>>>> andrew_schofield_jira@outlook.com> wrote:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>> I’ve been working to complete KIP-932 over the past few months and
> >>>>>>> discussions have quietened down.
> >>>>>>>
> >>>>>>> I’d like to open the voting for KIP-932:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Andrew
> >>>>>>
> >>>>
>
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Andrew Schofield <an...@live.com>.
Hi,
I have made a small update to the KIP as a result of testing the new
share consumer with client telemetry (KIP-714).
I’ve added telemetry metric names to the table of client metrics and
also updated the metric group names so that the resulting client metrics
sent to the broker have consistent names.
Thanks,
Andrew
> On 8 May 2024, at 12:51, Manikumar <ma...@gmail.com> wrote:
>
> Hi Andrew,
>
> Thanks for the KIP. Great write-up!
>
> +1 (binding)
>
> Thanks,
>
> On Wed, May 8, 2024 at 12:17 PM Satish Duggana <sa...@gmail.com> wrote:
>>
>> Hi Andrew,
>> Thanks for the nice KIP, it will allow other messaging use cases to be
>> onboarded to Kafka.
>>
>> +1 from me.
>>
>> Satish.
>>
>> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid> wrote:
>>>
>>> Hi, Andrew,
>>>
>>> Thanks for the KIP. +1
>>>
>>> Jun
>>>
>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <ed...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Andrew,
>>>>
>>>> +1 (binding)
>>>>
>>>> Edo
>>>>
>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
>>>> <ke...@cloudflare.com.invalid> wrote:
>>>>>
>>>>> Hi Andrew
>>>>>
>>>>> + 1 (Non-Binding)
>>>>>
>>>>> This will be great addition to Kafka
>>>>>
>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <ap...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Andrew,
>>>>>> Thanks for writing the KIP. This is indeed going to be a valuable
>>>> addition
>>>>>> to the Kafka, excited to see the KIP.
>>>>>>
>>>>>> + 1 (Non-Binding)
>>>>>>
>>>>>> Regards,
>>>>>> Apoorv Mittal
>>>>>> +44 7721681581
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
>>>>>> andrew_schofield_jira@outlook.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I’ve been working to complete KIP-932 over the past few months and
>>>>>>> discussions have quietened down.
>>>>>>>
>>>>>>> I’d like to open the voting for KIP-932:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>
>>>>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Manikumar <ma...@gmail.com>.
Hi Andrew,
Thanks for the KIP. Great write-up!
+1 (binding)
Thanks,
On Wed, May 8, 2024 at 12:17 PM Satish Duggana <sa...@gmail.com> wrote:
>
> Hi Andrew,
> Thanks for the nice KIP, it will allow other messaging use cases to be
> onboarded to Kafka.
>
> +1 from me.
>
> Satish.
>
> On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid> wrote:
> >
> > Hi, Andrew,
> >
> > Thanks for the KIP. +1
> >
> > Jun
> >
> > On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <ed...@gmail.com>
> > wrote:
> >
> > > Thanks Andrew,
> > >
> > > +1 (binding)
> > >
> > > Edo
> > >
> > > On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> > > <ke...@cloudflare.com.invalid> wrote:
> > > >
> > > > Hi Andrew
> > > >
> > > > + 1 (Non-Binding)
> > > >
> > > > This will be great addition to Kafka
> > > >
> > > > On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <ap...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Andrew,
> > > > > Thanks for writing the KIP. This is indeed going to be a valuable
> > > addition
> > > > > to the Kafka, excited to see the KIP.
> > > > >
> > > > > + 1 (Non-Binding)
> > > > >
> > > > > Regards,
> > > > > Apoorv Mittal
> > > > > +44 7721681581
> > > > >
> > > > >
> > > > > On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > > > > andrew_schofield_jira@outlook.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > > I’ve been working to complete KIP-932 over the past few months and
> > > > > > discussions have quietened down.
> > > > > >
> > > > > > I’d like to open the voting for KIP-932:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > > > > >
> > > > > > Thanks,
> > > > > > Andrew
> > > > >
> > >
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Satish Duggana <sa...@gmail.com>.
Hi Andrew,
Thanks for the nice KIP, it will allow other messaging use cases to be
onboarded to Kafka.
+1 from me.
Satish.
On Tue, 7 May 2024 at 03:41, Jun Rao <ju...@confluent.io.invalid> wrote:
>
> Hi, Andrew,
>
> Thanks for the KIP. +1
>
> Jun
>
> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <ed...@gmail.com>
> wrote:
>
> > Thanks Andrew,
> >
> > +1 (binding)
> >
> > Edo
> >
> > On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> > <ke...@cloudflare.com.invalid> wrote:
> > >
> > > Hi Andrew
> > >
> > > + 1 (Non-Binding)
> > >
> > > This will be great addition to Kafka
> > >
> > > On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <ap...@gmail.com>
> > > wrote:
> > >
> > > > Hi Andrew,
> > > > Thanks for writing the KIP. This is indeed going to be a valuable
> > addition
> > > > to the Kafka, excited to see the KIP.
> > > >
> > > > + 1 (Non-Binding)
> > > >
> > > > Regards,
> > > > Apoorv Mittal
> > > > +44 7721681581
> > > >
> > > >
> > > > On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > > > andrew_schofield_jira@outlook.com> wrote:
> > > >
> > > > > Hi,
> > > > > I’ve been working to complete KIP-932 over the past few months and
> > > > > discussions have quietened down.
> > > > >
> > > > > I’d like to open the voting for KIP-932:
> > > > >
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > >
> >
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Jun Rao <ju...@confluent.io.INVALID>.
Hi, Andrew,
Thanks for the KIP. +1
Jun
On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar <ed...@gmail.com>
wrote:
> Thanks Andrew,
>
> +1 (binding)
>
> Edo
>
> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> <ke...@cloudflare.com.invalid> wrote:
> >
> > Hi Andrew
> >
> > + 1 (Non-Binding)
> >
> > This will be great addition to Kafka
> >
> > On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <ap...@gmail.com>
> > wrote:
> >
> > > Hi Andrew,
> > > Thanks for writing the KIP. This is indeed going to be a valuable
> addition
> > > to the Kafka, excited to see the KIP.
> > >
> > > + 1 (Non-Binding)
> > >
> > > Regards,
> > > Apoorv Mittal
> > > +44 7721681581
> > >
> > >
> > > On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > > andrew_schofield_jira@outlook.com> wrote:
> > >
> > > > Hi,
> > > > I’ve been working to complete KIP-932 over the past few months and
> > > > discussions have quietened down.
> > > >
> > > > I’d like to open the voting for KIP-932:
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > > >
> > > > Thanks,
> > > > Andrew
> > >
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Edoardo Comar <ed...@gmail.com>.
Thanks Andrew,
+1 (binding)
Edo
On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
<ke...@cloudflare.com.invalid> wrote:
>
> Hi Andrew
>
> + 1 (Non-Binding)
>
> This will be great addition to Kafka
>
> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <ap...@gmail.com>
> wrote:
>
> > Hi Andrew,
> > Thanks for writing the KIP. This is indeed going to be a valuable addition
> > to the Kafka, excited to see the KIP.
> >
> > + 1 (Non-Binding)
> >
> > Regards,
> > Apoorv Mittal
> > +44 7721681581
> >
> >
> > On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > andrew_schofield_jira@outlook.com> wrote:
> >
> > > Hi,
> > > I’ve been working to complete KIP-932 over the past few months and
> > > discussions have quietened down.
> > >
> > > I’d like to open the voting for KIP-932:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > >
> > > Thanks,
> > > Andrew
> >
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Kenneth Eversole <ke...@cloudflare.com.INVALID>.
Hi Andrew
+ 1 (Non-Binding)
This will be great addition to Kafka
On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal <ap...@gmail.com>
wrote:
> Hi Andrew,
> Thanks for writing the KIP. This is indeed going to be a valuable addition
> to the Kafka, excited to see the KIP.
>
> + 1 (Non-Binding)
>
> Regards,
> Apoorv Mittal
> +44 7721681581
>
>
> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> andrew_schofield_jira@outlook.com> wrote:
>
> > Hi,
> > I’ve been working to complete KIP-932 over the past few months and
> > discussions have quietened down.
> >
> > I’d like to open the voting for KIP-932:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> >
> > Thanks,
> > Andrew
>
Re: [VOTE] KIP-932: Queues for Kafka
Posted by Apoorv Mittal <ap...@gmail.com>.
Hi Andrew,
Thanks for writing the KIP. This is indeed going to be a valuable addition
to the Kafka, excited to see the KIP.
+ 1 (Non-Binding)
Regards,
Apoorv Mittal
+44 7721681581
On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
andrew_schofield_jira@outlook.com> wrote:
> Hi,
> I’ve been working to complete KIP-932 over the past few months and
> discussions have quietened down.
>
> I’d like to open the voting for KIP-932:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>
> Thanks,
> Andrew
Re: [VOTE] KIP-932: Queues for Kafka
Posted by "Lianet M." <li...@gmail.com>.
Hi Andrew,
Happy to see this KIP unleashing the capabilities of Kafka, from the core
up, to elegantly bring in a new, long wanted consumption pattern.
+1 (non-binding)
Lianet
On Sun, Mar 17, 2024, 7:06 p.m. Andrew Schofield <
andrew_schofield_jira@outlook.com> wrote:
> Hi,
> I’ve been working to complete KIP-932 over the past few months and
> discussions have quietened down.
>
> I’d like to open the voting for KIP-932:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>
> Thanks,
> Andrew