You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "M. Manna" <ma...@gmail.com> on 2018/09/02 14:57:32 UTC
Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always
Round-Robin" Selection
Thanks for all your comments and taking it easy on me for my first KIP :)
I am trying to check if it's okay for us to start a vote on this? As per
some recent comment I'll change the name to RoundRobinPartitioner.
I'll need to put some effort in writing Scala tests etc. since I'm a novice
with Scala.
Please let me know your thoughts, and I'll update the status accordingly
(and start working on the JIRA once it's approved).
Regards,
On Fri, 31 Aug 2018, 10:22 M. Manna, <ma...@gmail.com> wrote:
> Yes I’m more than happy to change it to a more appropriate name.
>
> The issue with RoundRobinPatitoner is that the DefaultPartitioner already
> has a Round-Robin associated to it. But if community doesn’t mind the name,
> I don’t either.
>
> Thanks for reading the KIP btw.
>
> Regards,
>
> On Fri, 31 Aug 2018 at 05:47, Magesh Nandakumar <ma...@confluent.io>
> wrote:
>
>> +1 for this. The only small suggestion would be to possibly call this
>> RondRobinPartitioner which makes the intent obvious.
>>
>> On Thu, Aug 30, 2018 at 5:31 PM Stephen Powis <sp...@salesforce.com>
>> wrote:
>>
>> > Neat, this would be super helpful! I submitted this ages ago:
>> > https://issues.apache.org/jira/browse/KAFKA-3333
>> >
>> > On Fri, Aug 31, 2018 at 5:04 AM, Satish Duggana <
>> satish.duggana@gmail.com>
>> > wrote:
>> >
>> > > +including both dev and user mailing lists.
>> > >
>> > > Hi,
>> > > Thanks for the KIP.
>> > >
>> > > "* For us, the message keys represent some metadata which we use to
>> > either
>> > > ignore messages (if a loop-back to the sender), or log some
>> > information.*"
>> > >
>> > > Above statement was mentioned in the KIP about how key value is used.
>> I
>> > > guess the topic is not configured to be compacted and you do not want
>> to
>> > > have partitioning based on that key. IMHO, it qualifies more as a
>> header
>> > > than a key. What do you think about building records with a specific
>> > header
>> > > and consumers to execute the logic whether to process or ignore the
>> > > messages based on that header value.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > >
>> > > On Fri, Aug 31, 2018 at 1:32 AM, Satish Duggana <
>> > satish.duggana@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > > Thanks for the KIP.
>> > > >
>> > > > "* For us, the message keys represent some metadata which we use to
>> > > > either ignore messages (if a loop-back to the sender), or log some
>> > > > information.*"
>> > > >
>> > > > Above statement was mentioned in the KIP about how key value is
>> used. I
>> > > > guess the topic is not configured to be compacted and you do not
>> want
>> > to
>> > > > have partitioning based on that key. IMHO, it qualifies more as a
>> > header
>> > > > than a key. What do you think about building records with a specific
>> > > header
>> > > > and consumers to execute the logic whether to process or ignore the
>> > > > messages based on that header value.
>> > > >
>> > > > Thanks,
>> > > > Satish.
>> > > >
>> > > >
>> > > > On Fri, Aug 31, 2018 at 12:02 AM, M. Manna <ma...@gmail.com>
>> wrote:
>> > > >
>> > > >> Hi Harsha,
>> > > >>
>> > > >> thanks for reading the KIP.
>> > > >>
>> > > >> The intent is to use the DefaultPartitioner logic for round-robin
>> > > >> selection
>> > > >> of partition regardless of key type.
>> > > >>
>> > > >> Implementing Partitioner interface isn’t the issue here, you would
>> > have
>> > > to
>> > > >> do that anyway if you are implementing your own. But we also want
>> > this
>> > > to
>> > > >> be part of formal codebase.
>> > > >>
>> > > >> Regards,
>> > > >>
>> > > >> On Thu, 30 Aug 2018 at 16:58, Harsha <ka...@harsha.io> wrote:
>> > > >>
>> > > >> > Hi,
>> > > >> > Thanks for the KIP. I am trying to understand the intent of
>> > the
>> > > >> > KIP. Is the use case you specified can't be achieved by
>> > implementing
>> > > >> the
>> > > >> > Partitioner interface here?
>> > > >> > https://github.com/apache/kafka/blob/trunk/clients/src/main/
>> > > >> java/org/apache/kafka/clients/producer/Partitioner.java#L28
>> > > >> > .
>> > > >> > Use your custom partitioner to be configured in your producer
>> > clients.
>> > > >> >
>> > > >> > Thanks,
>> > > >> > Harsha
>> > > >> >
>> > > >> > On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote:
>> > > >> > > Hello,
>> > > >> > >
>> > > >> > > I opened a very simple KIP and there exists a JIRA for it.
>> > > >> > >
>> > > >> > > I would be grateful if any comments are available for action.
>> > > >> > >
>> > > >> > > Regards,
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always
Round-Robin" Selection
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks for the KIP.
I was initially wondering if we need to add this into Kafka code base
not. You could just go with a custom `Partitioner` implementation.
The demand of KAFKA-3333 seems not to be big actually. On the other
hand, it might be a generic use case that justifies to add to Kafka code
base. Also, it's not much code and thus easy to maintain.
About naming: `RoundRobinPartitioner` seems to be a good name.
`KeylessPartitioner` does not describe how partitions are selected. I
don't see any conflict with `DefaultPartitioner` -- it implements a
hybrid strategy depending if key is `null` or not (thus finding a name
better than `DefaultPartitioner` that describes the strategy seem to be
difficult).
Feel free to start a VOTE :) I don't expect this KIP to be controversial.
-Matthias
On 9/2/18 7:57 AM, M. Manna wrote:
> Thanks for all your comments and taking it easy on me for my first KIP :)
>
> I am trying to check if it's okay for us to start a vote on this? As per
> some recent comment I'll change the name to RoundRobinPartitioner.
>
> I'll need to put some effort in writing Scala tests etc. since I'm a novice
> with Scala.
>
> Please let me know your thoughts, and I'll update the status accordingly
> (and start working on the JIRA once it's approved).
>
> Regards,
>
> On Fri, 31 Aug 2018, 10:22 M. Manna, <ma...@gmail.com> wrote:
>
>> Yes I’m more than happy to change it to a more appropriate name.
>>
>> The issue with RoundRobinPatitoner is that the DefaultPartitioner already
>> has a Round-Robin associated to it. But if community doesn’t mind the name,
>> I don’t either.
>>
>> Thanks for reading the KIP btw.
>>
>> Regards,
>>
>> On Fri, 31 Aug 2018 at 05:47, Magesh Nandakumar <ma...@confluent.io>
>> wrote:
>>
>>> +1 for this. The only small suggestion would be to possibly call this
>>> RondRobinPartitioner which makes the intent obvious.
>>>
>>> On Thu, Aug 30, 2018 at 5:31 PM Stephen Powis <sp...@salesforce.com>
>>> wrote:
>>>
>>>> Neat, this would be super helpful! I submitted this ages ago:
>>>> https://issues.apache.org/jira/browse/KAFKA-3333
>>>>
>>>> On Fri, Aug 31, 2018 at 5:04 AM, Satish Duggana <
>>> satish.duggana@gmail.com>
>>>> wrote:
>>>>
>>>>> +including both dev and user mailing lists.
>>>>>
>>>>> Hi,
>>>>> Thanks for the KIP.
>>>>>
>>>>> "* For us, the message keys represent some metadata which we use to
>>>> either
>>>>> ignore messages (if a loop-back to the sender), or log some
>>>> information.*"
>>>>>
>>>>> Above statement was mentioned in the KIP about how key value is used.
>>> I
>>>>> guess the topic is not configured to be compacted and you do not want
>>> to
>>>>> have partitioning based on that key. IMHO, it qualifies more as a
>>> header
>>>>> than a key. What do you think about building records with a specific
>>>> header
>>>>> and consumers to execute the logic whether to process or ignore the
>>>>> messages based on that header value.
>>>>>
>>>>> Thanks,
>>>>> Satish.
>>>>>
>>>>>
>>>>> On Fri, Aug 31, 2018 at 1:32 AM, Satish Duggana <
>>>> satish.duggana@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Thanks for the KIP.
>>>>>>
>>>>>> "* For us, the message keys represent some metadata which we use to
>>>>>> either ignore messages (if a loop-back to the sender), or log some
>>>>>> information.*"
>>>>>>
>>>>>> Above statement was mentioned in the KIP about how key value is
>>> used. I
>>>>>> guess the topic is not configured to be compacted and you do not
>>> want
>>>> to
>>>>>> have partitioning based on that key. IMHO, it qualifies more as a
>>>> header
>>>>>> than a key. What do you think about building records with a specific
>>>>> header
>>>>>> and consumers to execute the logic whether to process or ignore the
>>>>>> messages based on that header value.
>>>>>>
>>>>>> Thanks,
>>>>>> Satish.
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 31, 2018 at 12:02 AM, M. Manna <ma...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> Hi Harsha,
>>>>>>>
>>>>>>> thanks for reading the KIP.
>>>>>>>
>>>>>>> The intent is to use the DefaultPartitioner logic for round-robin
>>>>>>> selection
>>>>>>> of partition regardless of key type.
>>>>>>>
>>>>>>> Implementing Partitioner interface isn’t the issue here, you would
>>>> have
>>>>> to
>>>>>>> do that anyway if you are implementing your own. But we also want
>>>> this
>>>>> to
>>>>>>> be part of formal codebase.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> On Thu, 30 Aug 2018 at 16:58, Harsha <ka...@harsha.io> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Thanks for the KIP. I am trying to understand the intent of
>>>> the
>>>>>>>> KIP. Is the use case you specified can't be achieved by
>>>> implementing
>>>>>>> the
>>>>>>>> Partitioner interface here?
>>>>>>>> https://github.com/apache/kafka/blob/trunk/clients/src/main/
>>>>>>> java/org/apache/kafka/clients/producer/Partitioner.java#L28
>>>>>>>> .
>>>>>>>> Use your custom partitioner to be configured in your producer
>>>> clients.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Harsha
>>>>>>>>
>>>>>>>> On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I opened a very simple KIP and there exists a JIRA for it.
>>>>>>>>>
>>>>>>>>> I would be grateful if any comments are available for action.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>