You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Feiyan Yu <db...@gmail.com> on 2022/03/26 06:56:05 UTC
Potential bugs in resizing kafka thread
Howdy!
I found that the method of resizing fetching thread had one potential bug
which lead to imbalanced partitions distribution for each thread after
changing thread number on dynamic configuration.
To figure it out, I added a primitive unit test method "testResize" in
"AbstractFetcherThreadTest" to simulate the replica fetchers resizing
process from 10 threads to 60 threads with originally 10 topics and 100
partitions each. I designed the test to show that after the resizing
process, all partitions should be redistributed correctly based on the new
thread number. However, the test failed because when I tried to compare the
"fetcherThreadMap" with the fetcherId for each topic-partition, the
fetcherId mismatched! The unit test I added is in this commit (
https://github.com/yufeiyan1220/kafka/commit/eb99b7499b416cdeb44c2ccd3ea55a1e38ea3d60),
and the standard output of the unit test showed in attachment.
I doubt that maybe it is because the method "addFetcherForPartitions" which
maybe adds some new fetchers to "fetcherThreadMap" called in the block of
iterating the "fetcherThreadMap", and the iterator ignore some fetchers,
which leads to some of the fetchers remain their topic partitions. And it
leads to the imbalanced partition distribution pattern in resizing.
To solve this issue I make a mutable map to store all partitions and its
fetch offset, and then add it back once out of the iteration. I make
another commit (
https://github.com/yufeiyan1220/kafka/commit/0a793dfca2ab9b8e8b45ba5693359960f3e306eb),
and the new resize method passed the unit test.
I'm not sure whether it is an issue that Kafka Community need to fix. But
for me, it affects the fetching efficiency when I try to deploy Kafka cross
regions with high network latency.
I'd really appreciate to hear from you!
Re: Potential bugs in resizing kafka thread
Posted by Feiyan Yu <db...@gmail.com>.
I fixed some ambiguous expression in the output of unit test, and also open
a bug in JIRA <https://issues.apache.org/jira/browse/KAFKA-13772> with a
pull request <https://issues.apache.org/jira/browse/KAFKA-13772>
Thank you for your help and suggestions!
Feiyan Yu <db...@gmail.com> 于2022年3月26日周六 16:18写道：
> Thank you, I will make it an issue.
>
> Luke Chen <sh...@gmail.com> 于2022年3月26日周六 16:03写道：
>
>> Hi Feiyan,
>>
>> It did look like a bug. Could you open a bug in JIRA here
>> <
>> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13735?filter=allopenissues
>> >
>> ?
>> One thing I don't fully understand from the unit test is that, you have
>> output "error distributions: $tpsWithoutResize".
>> I thought the bug is about the unevenly partition distribution after
>> resizing fetch thread, what do you mean by "error distribution"?
>> Could you please also elaborate in JIRA ticket?
>>
>> Also, welcome to submit a PR to it, since you've already had fix and test
>> ready! :)
>>
>> Thank you.
>> Luke
>>
>>
>>
>> On Sat, Mar 26, 2022 at 2:56 PM Feiyan Yu <db...@gmail.com> wrote:
>>
>> > Howdy!
>> >
>> > I found that the method of resizing fetching thread had one potential
>> bug
>> > which lead to imbalanced partitions distribution for each thread after
>> > changing thread number on dynamic configuration.
>> >
>> > To figure it out, I added a primitive unit test method "testResize" in
>> > "AbstractFetcherThreadTest" to simulate the replica fetchers resizing
>> > process from 10 threads to 60 threads with originally 10 topics and 100
>> > partitions each. I designed the test to show that after the resizing
>> > process, all partitions should be redistributed correctly based on the
>> new
>> > thread number. However, the test failed because when I tried to compare
>> the
>> > "fetcherThreadMap" with the fetcherId for each topic-partition, the
>> > fetcherId mismatched! The unit test I added is in this commit (
>> >
>> https://github.com/yufeiyan1220/kafka/commit/eb99b7499b416cdeb44c2ccd3ea55a1e38ea3d60
>> ),
>> > and the standard output of the unit test showed in attachment.
>> >
>> > I doubt that maybe it is because the method "addFetcherForPartitions"
>> > which maybe adds some new fetchers to "fetcherThreadMap" called in the
>> > block of iterating the "fetcherThreadMap", and the iterator ignore some
>> > fetchers, which leads to some of the fetchers remain their topic
>> > partitions. And it leads to the imbalanced partition distribution
>> pattern
>> > in resizing.
>> >
>> > To solve this issue I make a mutable map to store all partitions and its
>> > fetch offset, and then add it back once out of the iteration. I make
>> > another commit (
>> >
>> https://github.com/yufeiyan1220/kafka/commit/0a793dfca2ab9b8e8b45ba5693359960f3e306eb
>> ),
>> > and the new resize method passed the unit test.
>> >
>> > I'm not sure whether it is an issue that Kafka Community need to fix.
>> But
>> > for me, it affects the fetching efficiency when I try to deploy Kafka
>> cross
>> > regions with high network latency.
>> >
>> > I'd really appreciate to hear from you!
>> >
>> >
>>
>
Re: Potential bugs in resizing kafka thread
Posted by Feiyan Yu <db...@gmail.com>.
Thank you, I will make it an issue.
Luke Chen <sh...@gmail.com> 于2022年3月26日周六 16:03写道：
> Hi Feiyan,
>
> It did look like a bug. Could you open a bug in JIRA here
> <
> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13735?filter=allopenissues
> >
> ?
> One thing I don't fully understand from the unit test is that, you have
> output "error distributions: $tpsWithoutResize".
> I thought the bug is about the unevenly partition distribution after
> resizing fetch thread, what do you mean by "error distribution"?
> Could you please also elaborate in JIRA ticket?
>
> Also, welcome to submit a PR to it, since you've already had fix and test
> ready! :)
>
> Thank you.
> Luke
>
>
>
> On Sat, Mar 26, 2022 at 2:56 PM Feiyan Yu <db...@gmail.com> wrote:
>
> > Howdy!
> >
> > I found that the method of resizing fetching thread had one potential bug
> > which lead to imbalanced partitions distribution for each thread after
> > changing thread number on dynamic configuration.
> >
> > To figure it out, I added a primitive unit test method "testResize" in
> > "AbstractFetcherThreadTest" to simulate the replica fetchers resizing
> > process from 10 threads to 60 threads with originally 10 topics and 100
> > partitions each. I designed the test to show that after the resizing
> > process, all partitions should be redistributed correctly based on the
> new
> > thread number. However, the test failed because when I tried to compare
> the
> > "fetcherThreadMap" with the fetcherId for each topic-partition, the
> > fetcherId mismatched! The unit test I added is in this commit (
> >
> https://github.com/yufeiyan1220/kafka/commit/eb99b7499b416cdeb44c2ccd3ea55a1e38ea3d60
> ),
> > and the standard output of the unit test showed in attachment.
> >
> > I doubt that maybe it is because the method "addFetcherForPartitions"
> > which maybe adds some new fetchers to "fetcherThreadMap" called in the
> > block of iterating the "fetcherThreadMap", and the iterator ignore some
> > fetchers, which leads to some of the fetchers remain their topic
> > partitions. And it leads to the imbalanced partition distribution pattern
> > in resizing.
> >
> > To solve this issue I make a mutable map to store all partitions and its
> > fetch offset, and then add it back once out of the iteration. I make
> > another commit (
> >
> https://github.com/yufeiyan1220/kafka/commit/0a793dfca2ab9b8e8b45ba5693359960f3e306eb
> ),
> > and the new resize method passed the unit test.
> >
> > I'm not sure whether it is an issue that Kafka Community need to fix. But
> > for me, it affects the fetching efficiency when I try to deploy Kafka
> cross
> > regions with high network latency.
> >
> > I'd really appreciate to hear from you!
> >
> >
>
Re: Potential bugs in resizing kafka thread
Posted by Luke Chen <sh...@gmail.com>.
Hi Feiyan,
It did look like a bug. Could you open a bug in JIRA here
<https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13735?filter=allopenissues>
?
One thing I don't fully understand from the unit test is that, you have
output "error distributions: $tpsWithoutResize".
I thought the bug is about the unevenly partition distribution after
resizing fetch thread, what do you mean by "error distribution"?
Could you please also elaborate in JIRA ticket?
Also, welcome to submit a PR to it, since you've already had fix and test
ready! :)
Thank you.
Luke
On Sat, Mar 26, 2022 at 2:56 PM Feiyan Yu <db...@gmail.com> wrote:
> Howdy!
>
> I found that the method of resizing fetching thread had one potential bug
> which lead to imbalanced partitions distribution for each thread after
> changing thread number on dynamic configuration.
>
> To figure it out, I added a primitive unit test method "testResize" in
> "AbstractFetcherThreadTest" to simulate the replica fetchers resizing
> process from 10 threads to 60 threads with originally 10 topics and 100
> partitions each. I designed the test to show that after the resizing
> process, all partitions should be redistributed correctly based on the new
> thread number. However, the test failed because when I tried to compare the
> "fetcherThreadMap" with the fetcherId for each topic-partition, the
> fetcherId mismatched! The unit test I added is in this commit (
> https://github.com/yufeiyan1220/kafka/commit/eb99b7499b416cdeb44c2ccd3ea55a1e38ea3d60),
> and the standard output of the unit test showed in attachment.
>
> I doubt that maybe it is because the method "addFetcherForPartitions"
> which maybe adds some new fetchers to "fetcherThreadMap" called in the
> block of iterating the "fetcherThreadMap", and the iterator ignore some
> fetchers, which leads to some of the fetchers remain their topic
> partitions. And it leads to the imbalanced partition distribution pattern
> in resizing.
>
> To solve this issue I make a mutable map to store all partitions and its
> fetch offset, and then add it back once out of the iteration. I make
> another commit (
> https://github.com/yufeiyan1220/kafka/commit/0a793dfca2ab9b8e8b45ba5693359960f3e306eb),
> and the new resize method passed the unit test.
>
> I'm not sure whether it is an issue that Kafka Community need to fix. But
> for me, it affects the fetching efficiency when I try to deploy Kafka cross
> regions with high network latency.
>
> I'd really appreciate to hear from you!
>
>