You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Gary Russell (Jira)" <ji...@apache.org> on 2021/07/08 21:56:00 UTC
[jira] [Comment Edited] (KAFKA-13054) Unexpected Polling Behavior
[ https://issues.apache.org/jira/browse/KAFKA-13054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377606#comment-17377606 ]
Gary Russell edited comment on KAFKA-13054 at 7/8/21, 9:55 PM:
---------------------------------------------------------------
Yes, after increasing the partitions, I get 2000 on each poll and 1000 on the last.
was (Author: grussell):
Yes, after increasing the partitions, I get 2000 on each poll and 1000 on the last; you can close this.
> Unexpected Polling Behavior
> ---------------------------
>
> Key: KAFKA-13054
> URL: https://issues.apache.org/jira/browse/KAFKA-13054
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 2.8.0
> Reporter: Gary Russell
> Priority: Major
>
> Given a topic with 10 partitions and 9000 records (evenly distributed) and
> {noformat}
> max.poll.records: 2000
> fetch.max.bytes = 52428800
> fetch.max.wait.ms = 10000
> fetch.min.bytes = 10240000
> {noformat}
> We see odd results
> {code:java}
> log.info("" + recs.count() + "\n"
> + recs.partitions().stream()
> .map(part -> "" + part.partition()
> + "(" + recs.records(part).size() + ")")
> .collect(Collectors.toList()));
> {code}
> Result:
> {noformat}
> 2021-07-08 15:04:11.131 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 2000
> [1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201), 6(191)]
> 2021-07-08 15:04:11.137 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 10
> [6(10)]
> 2021-07-08 15:04:21.170 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1809
> [1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201)]
> 2021-07-08 15:04:21.214 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 2000
> [1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201), 6(191)]
> 2021-07-08 15:04:21.215 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 10
> [6(10)]
> 2021-07-08 15:04:31.248 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1809
> [1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201)]
> 2021-07-08 15:04:41.267 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1083
> [1(27), 0(87), 5(189), 4(93), 3(114), 2(129), 9(108), 8(93), 7(42), 6(201)]
> 2021-07-08 15:04:51.276 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 201
> [6(201)]
> 2021-07-08 15:05:01.279 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 78
> [6(78)]
> {noformat}
> I can understand the second poll returning immediately (presumably over-fetch from the previous poll) but why does the third poll take 10 seconds (fetch.max.wait)? The fourth and fifth polls are like the first and second and return almost immediately, but the sixth once again takes 10 seconds and returns a partial result.
> Is there a document that explains the fetch mechanism that might explain this behavior?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)