You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "sanghyeok An (Jira)" <ji...@apache.org> on 2026/04/09 15:33:00 UTC

[jira] [Commented] (KAFKA-20429) Review async consumer single background queue design

    [ https://issues.apache.org/jira/browse/KAFKA-20429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18072353#comment-18072353 ] 

sanghyeok An commented on KAFKA-20429:
--------------------------------------

Hi, Lianet!

If you are not working on this, may I look at this issue?

> Review async consumer single background queue design
> ----------------------------------------------------
>
>                 Key: KAFKA-20429
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20429
>             Project: Kafka
>          Issue Type: Task
>          Components: clients, consumer
>            Reporter: Lianet Magrans
>            Priority: Major
>             Fix For: 4.4.0
>
>
> The async consumer relies on a single queue to pass events from the background to the app thread. But this queue mixes different kinds of events (coord errors, metadata errors, callbacks, internal updates events like applyAssignment).
> This makes it problematic and complicated to use. When processingBackgroundEvents from the app thread, we need to filter-out events depending on what we're processing background events for. E.g., : 
>  * unsubscribe cares about callbacks only, needs to ignore all others (error prone as new events are added, see https://issues.apache.org/jira/browse/KAFKA-20428)
>  * poll cares about all, but actually could maybe benefit from the ability to process errors and callbacks at different points of the poll loop (see https://issues.apache.org/jira/browse/KAFKA-20397) 
> I wonder if re-designing the background events queue would simplify it's usage from the AsyncKafkaConsumer, and avoid continuously falling into errors when new events are added. Also we could have a simpler/clearer intention from the app thread, consuming from each func what it needs (vs consume all and filter out).
> One option that comes to mind is to have separate queues for errors, callbacks, internal ops. So app thread should be simpler (explicit intention when processing events from a queue), but to review in detail to see if we could get a simpler design, less error prone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)