You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Kirk True (Jira)" <ji...@apache.org> on 2024/01/09 00:08:00 UTC

[jira] [Updated] (KAFKA-15173) Consumer event queues should be bounded

     [ https://issues.apache.org/jira/browse/KAFKA-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kirk True updated KAFKA-15173:
------------------------------
    Fix Version/s: 3.8.0
                       (was: 4.0.0)

> Consumer event queues should be bounded
> ---------------------------------------
>
>                 Key: KAFKA-15173
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15173
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, consumer
>            Reporter: Philip Nee
>            Assignee: Kirk True
>            Priority: Major
>              Labels: consumer-threading-refactor
>             Fix For: 3.8.0
>
>
> The async consumer uses ApplicationEventQueue and BackgroundEventQueue to facilitate message passing between the application thread and the background thread.  The current implementation is boundless, which can potentially cause OOM and other performance-related issues.
> I think the queues need a finite bound, and we need to decide how to handle the situation when the bound is reached.  In particular, I would like to answer these questions:
>  
>  # What should the upper limit be for both queues: Can this be a configurable, memory-based bound? Or just an arbitrary number of events as the bound.
>  # What should happen when the application event queue is filled up?  It seems like we should introduce a new exception type and notify the user that the consumer is full.
>  # What should happen when the background event queue is filled up? This seems less likely to happen, but I imagine it could happen when the user stops polling the consumer, causing the queue to be filled.
>  # Is it necessary to introduce a public configuration for the queue? I think initially we would select an arbitrary constant number and see the community feedback to make a forward plan accordingly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)