[jira] [Assigned] (KAFKA-14968) Consumer Notification Callback

["Hendrik Jan van Randen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-14968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hendrik Jan van Randen reassigned KAFKA-14968:
----------------------------------------------

    Assignee: Hendrik Jan van Randen

> Consumer Notification Callback
> ------------------------------
>
>                 Key: KAFKA-14968
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14968
>             Project: Kafka
>          Issue Type: New Feature
>          Components: consumer
>            Reporter: Hendrik Jan van Randen
>            Assignee: Hendrik Jan van Randen
>            Priority: Major
>
> We want to use Kafka for all the communication between our services. 
> We have many high frequency topics (many messages per second), for which the Kafka mechanism of polling by consumers is efficient and suitable.
> However, we also have low frequency topics (no or few messages per day). If such a low frequency message is produced, all consumers should consume it immediately. But it would be inefficient if all those consumers are polling continuously with a high frequency (e.g. each second), while no or few such messages are produced each day. Most efficient would be if for such a low frequency topic the consumers can register a callback (e.g. a REST endpoint) with Kafka, and Kafka calls that callback when a message is produced. Either supplying the message in the call (pro: consumer can handle it immediately), or just to notify the consumer that it should do a one time poll (pro: same message transfer mechanism as everywhere).
> I just talked this over with Ryan Corrigan of Confluent.
> In the coming weeks I will supply more information about the urgency of this issue.
> I will be present at the Kafka Summit in London on May 16 & 17, 2023. Who of the Confluent technical staff, or Kafka committers, is present at the Kafka Summit to investigate possible solution directions for this?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)