[jira] [Assigned] (KAFKA-20780) syncKafkaConsumer does not keep a fetch request pending on the broker, causing latency and throughput regressions vs the classic consumer

["Lianet Magrans (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lianet Magrans reassigned KAFKA-20780:
--------------------------------------

    Assignee: Lianet Magrans

> syncKafkaConsumer does not keep a fetch request pending on the broker, causing latency and throughput regressions vs the classic consumer
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-20780
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20780
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: Blaž Šnuderl
>            Assignee: Lianet Magrans
>            Priority: Major
>
> When testing out kip-848 we see worse end to end latencies on realtime topics.
> The AsyncKafkaConsumer (KIP-848, group.protocol=consumer) leaves gaps where no fetch request is pending on the broker. The classic KafkaConsumer keeps one in flight at all times. On the same low-throughput topic (~1 msg/s), the classic consumer sees ~3.2 fetches/s and ~28ms latency, while the async consumer sees ~0.47 fetches/s and ~2300ms latency — an ~80x latency regression from switching the group protocol alone.\{{ConsumerNetworkThread.runOnce()}} sleeps for \{{maximumTimeToWait()}} between iterations with no fetch pending, so messages arriving in the gap wait until the next wakeup.
> {\{AsyncKafkaConsumer.checkInflightPoll()}} does the \{{inflightPoll == null}} check before \{{maybeClearCurrentInflightPoll()}}. So on even iterations \{{inflightPoll}} is still non-null, no new fetch event is submitted, and \{{pollForFetches}} blocks the full timeout. Net: one fetch per ~2.1s.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)