You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Gary Russell (Jira)" <ji...@apache.org> on 2021/07/08 21:30:00 UTC
[jira] [Created] (KAFKA-13054) Unexpected Polling Behavior
Gary Russell created KAFKA-13054:
------------------------------------
Summary: Unexpected Polling Behavior
Key: KAFKA-13054
URL: https://issues.apache.org/jira/browse/KAFKA-13054
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 2.8.0
Reporter: Gary Russell
Given a topic with 10 partitions and 9000 records (evenly distributed) and
{noformat}
max.poll.records: 2000
fetch.max.bytes = 52428800
fetch.max.wait.ms = 10000
fetch.min.bytes = 10240000
{noformat}
We see odd results
{code:java}
log.info("" + recs.count() + "\n"
+ recs.partitions().stream()
.map(part -> "" + part.partition()
+ "(" + recs.records(part).size() + ")")
.collect(Collectors.toList()));
{code}
Result:
{noformat}
2021-07-08 15:04:11.131 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 2000
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201), 6(191)]
2021-07-08 15:04:11.137 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 10
[6(10)]
2021-07-08 15:04:21.170 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1809
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201)]
2021-07-08 15:04:21.214 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 2000
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201), 6(191)]
2021-07-08 15:04:21.215 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 10
[6(10)]
2021-07-08 15:04:31.248 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1809
[1(201), 0(201), 5(201), 4(201), 3(201), 2(201), 9(201), 8(201), 7(201)]
2021-07-08 15:04:41.267 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 1083
[1(27), 0(87), 5(189), 4(93), 3(114), 2(129), 9(108), 8(93), 7(42), 6(201)]
2021-07-08 15:04:51.276 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 201
[6(201)]
2021-07-08 15:05:01.279 INFO 45792 --- [o68201599-0-C-1] com.example.demo.So68201599Application : 78
[6(78)]
{noformat}
I can understand the second poll returning immediately (presumably over-fetch from the previous poll) but why does the third poll take 10 seconds (fetch.max.wait)? The fourth and fifth polls are like the first and second and return almost immediately, but the sixth once again takes 10 seconds and returns a partial result.
Is there a document that explains the fetch mechanism that might explain this behavior?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)