[jira] [Created] (KAFKA-20429) Review async consumer single background queue design

["Lianet Magrans (Jira)" <ji...@apache.org>]

Lianet Magrans created KAFKA-20429:
--------------------------------------

             Summary: Review async consumer single background queue design
                 Key: KAFKA-20429
                 URL: https://issues.apache.org/jira/browse/KAFKA-20429
             Project: Kafka
          Issue Type: Task
          Components: clients, consumer
            Reporter: Lianet Magrans


The async consumer relies on a single queue to pass events from the background to the app thread. But this queue mixes different kinds of events (coord errors, metadata errors, callbacks, internal updates events like applyAssignment).

This makes it problematic and complicated to use. When processingBackgroundEvents from the app thread, we need to filter-out events depending on what we're processing background events for. E.g., : 
 * unsubscribe cares about callbacks only, needs to ignore all others (error prone as new events are added, see https://issues.apache.org/jira/browse/KAFKA-20428)
 * poll cares about all, but actually could maybe benefit from the ability to process errors and callbacks at different points of the poll loop (see https://issues.apache.org/jira/browse/KAFKA-20397) 

I wonder if re-designing the background events queue would simplify it's usage from the AsyncKafkaConsumer, and avoid continuously falling into errors when new events are added. Also we could have a simpler/clearer intention from the app thread, consuming from each func what it needs (vs consume all and filter out).

One option that comes to mind is to have separate queues for errors, callbacks, internal ops. So app thread should be simpler (explicit intention when processing events from a queue), but to review in detail to see if we could get a simpler design, less error prone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)