You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/09/30 11:13:15 UTC
[GitHub] [kafka] mathieu-amblard opened a new pull request, #12705: KAFKA-14269: client-side, add TopicRoundRobinAssignor as a possible partition assignment strategy
mathieu-amblard opened a new pull request, #12705:
URL: https://github.com/apache/kafka/pull/12705
Add a new partition assignment strategy that assigns all partitions of a topic to exactly one consumer.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #12705: KAFKA-14269: client-side, add TopicRoundRobinAssignor as a possible partition assignment strategy
Posted by GitBox <gi...@apache.org>.
dajac commented on PR #12705:
URL: https://github.com/apache/kafka/pull/12705#issuecomment-1263546176
@mathieu-amblard Thanks for the PR. Could you elaborate a bit more on the motivation for this new assignor? I am not sure that I get it at the moment. What are the benefits of doing this?
> I would like to have a Topic Round Robin Assignor in order to assign all partitions of same topic to exactly one consumer, therefore I will be able to continue the consumption of one topic even if one failed.
When a consumer fails, all its partitions should be reassigned to other consumers in the group. This should not impact the other topics. Could you elaborate a bit more on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on pull request #12705: KAFKA-14269: client-side, add TopicRoundRobinAssignor as a possible partition assignment strategy
Posted by GitBox <gi...@apache.org>.
dajac commented on PR #12705:
URL: https://github.com/apache/kafka/pull/12705#issuecomment-1263778542
@mathieu-amblard Thanks for the explanation! That's an interesting approach. Is there a reason why you have multiple partitions per topic? That would be another way to provide the same guarantees if I understood your explanation correctly.
Regarding the assignor, it looks like a really application specific case so I am not sure that adding this into Apache Kafka is necessary at the moment. We have always tried to limit our assignors to the most common ones. Let's see what other community members think.
If you want to proceed with this, you'll have to do a KIP for it. The process is [here](https://cwiki.apache.org/confluence/display/kafka/kafka+improvement+proposals) in case you need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mathieu-amblard commented on pull request #12705: KAFKA-14269: client-side, add TopicRoundRobinAssignor as a possible partition assignment strategy
Posted by GitBox <gi...@apache.org>.
mathieu-amblard commented on PR #12705:
URL: https://github.com/apache/kafka/pull/12705#issuecomment-1263687207
@dajac Thanks for your reply and sorry if I was unclear in my description.
I forgot to mention something very important in my description, I would like to be thread safe as much as possible.
I also would like to have multiple consumers (1 thread per consumer) for performance purpose.
For business purposes, each topic contains only on type of object.
Most of them must should not diverge too much to have a good data consistency (ie. we should not continue to consume one partition of one topic if one partition failed).
Combining these hypothesis and requirements, I though that having as many consumers as topics is the best option if each partitions of a topic is assign to only one consumer.
-------------------------------------
**First requirement : be thread safe as much as possible**
If I use one of the four existing assignor, I always get mixed partitions assigned to a topic (ie. a partition of different topic) and so records of a same topic can be proceed concurrently.
Let me take an example.
Suppose, there are 2 consumers `C0` and `C1`, and two topics `t0` and `t1`, and each has 3 partitions, resulting in partitions `t0p0`, `t0p1`, `t0p2`, `tp1p0`, `t1p1` and `t1p2`.
If I use the standard RoundRobinAssignor, the assignement will be :
`C0: [t0p0, t0p2, t1p1]`
`C1: [t0p1, t1p0, t1p2]`
In that case, I must ensure that the processes executed when I received records from `t0` and `t1` must be thread safe because they can came from two different threads `C0` and `C1` in parallel.
If I use the TopicRoundRobinAssignor, the assignement will be :
`C0: [t0p0, t0p1, t0p2]`
`C1: [t1p0, t1p1, t1p2]`
In that case, each record record of a same topic is processed in the same thread so there are less risks of concurrency.
--------------------------------------
**Second requirement : topic should not diverge too much (keep data consistency)**
Again, if I use one of the four existing assignor, I always get mixed partitions assigned to a topic (ie. a partition of different topic).
If one of the consumer fails, I kept in memory which partition failed in order to not propagate the "corrupted" record.
Otherwise, after rebalancing, when the "corrupted" will be consumed by an other consumer, it will be stopped also, and so on, and so on...
Let me take the same example.
Suppose, there are 2 consumers `C0` and `C1`, and two topics `t0` and `t1`, and each has 3 partitions, resulting in partitions `t0p0`, `t0p1`, `t0p2`, `tp1p0`, `t1p1` and `t1p2`.
If I use the standard RoundRobinAssignor, the assignement will be :
`C0: [t0p0, t0p2, t1p1]`
`C1: [t0p1, t1p0, t1p2]`
If `C0` fails, it will be stopped and the new assignement will be :
`C1: [t0p1, t1p0, t1p2, t0p0, t0p2, t1p1]`
As I kept in memory that I do not want to consume `t0p0`, `t0p2`, `t1p1`, only records from `t0p1`, `t1p0`, `t1p2` will continue to be consumed.
Therefore, if I do not fix the issue quickly about the "corrupted" record, the data from each topic will diverge (some records will be consumed, some other not).
If I use the standard `TopicRoundRobinAssignor`, the assignment will be :
`C0: [t0p0, t0p1, t0p2]`
`C1: [t1p0, t1p1, t1p2]`
If `C0` fails, it will be stopped and the new assignment will be :
`C1: [t0p0, t0p1, t0p2, t1p0, t1p1, t1p2]`
As I kept in memory that I do not want to consume `t0p0`, `t0p1`, `t0p2`, only records from `t0p1`, `t0p1`, `t0p2` will continue to be consumed.
Therefore, I continue to consume all records of a same topic and my data stay consistent as much as possible.
------------------------------------------------
I hope this is more clear, this is not really simple to explain... Maybe my requirements are too specific and nobody else have the needs...
I remain of course at your disposal for any questions or for more explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mathieu-amblard closed pull request #12705: KAFKA-14269: client-side, add TopicRoundRobinAssignor as a possible partition assignment strategy
Posted by GitBox <gi...@apache.org>.
mathieu-amblard closed pull request #12705: KAFKA-14269: client-side, add TopicRoundRobinAssignor as a possible partition assignment strategy
URL: https://github.com/apache/kafka/pull/12705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mathieu-amblard commented on pull request #12705: KAFKA-14269: client-side, add TopicRoundRobinAssignor as a possible partition assignment strategy
Posted by GitBox <gi...@apache.org>.
mathieu-amblard commented on PR #12705:
URL: https://github.com/apache/kafka/pull/12705#issuecomment-1264951061
@dajac Thanks for your reply !
We have multiple partitions per topic because, we can have multiple pods running the same application. With the Topic Round Robin Assignor, the partition are uniformly balanced to each pod.
I understand that seems to be a specific case, I was asking myself if there is a better approach as there is not a such assignor in Kafka but I have not found yet. That's why I have made this proposal, I really wanted to challenge it and maybe it can help someone else...
So sure, let's see what the community members think, thanks for the information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org