You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "sanghyeok An (Jira)" <ji...@apache.org> on 2026/04/09 15:33:00 UTC
[jira] [Commented] (KAFKA-20429) Review async consumer single background queue design
[ https://issues.apache.org/jira/browse/KAFKA-20429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18072353#comment-18072353 ]
sanghyeok An commented on KAFKA-20429:
--------------------------------------
Hi, Lianet!
If you are not working on this, may I look at this issue?
> Review async consumer single background queue design
> ----------------------------------------------------
>
> Key: KAFKA-20429
> URL: https://issues.apache.org/jira/browse/KAFKA-20429
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
> Reporter: Lianet Magrans
> Priority: Major
> Fix For: 4.4.0
>
>
> The async consumer relies on a single queue to pass events from the background to the app thread. But this queue mixes different kinds of events (coord errors, metadata errors, callbacks, internal updates events like applyAssignment).
> This makes it problematic and complicated to use. When processingBackgroundEvents from the app thread, we need to filter-out events depending on what we're processing background events for. E.g., :
> * unsubscribe cares about callbacks only, needs to ignore all others (error prone as new events are added, see https://issues.apache.org/jira/browse/KAFKA-20428)
> * poll cares about all, but actually could maybe benefit from the ability to process errors and callbacks at different points of the poll loop (see https://issues.apache.org/jira/browse/KAFKA-20397)
> I wonder if re-designing the background events queue would simplify it's usage from the AsyncKafkaConsumer, and avoid continuously falling into errors when new events are added. Also we could have a simpler/clearer intention from the app thread, consuming from each func what it needs (vs consume all and filter out).
> One option that comes to mind is to have separate queues for errors, callbacks, internal ops. So app thread should be simpler (explicit intention when processing events from a queue), but to review in detail to see if we could get a simpler design, less error prone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)