[jira] [Created] (KAFKA-14968) Consumer Notification Callback

["Hendrik Jan van Randen (Jira)" <ji...@apache.org>]

Hendrik Jan van Randen created KAFKA-14968:
----------------------------------------------

             Summary: Consumer Notification Callback
                 Key: KAFKA-14968
                 URL: https://issues.apache.org/jira/browse/KAFKA-14968
             Project: Kafka
          Issue Type: New Feature
          Components: consumer
            Reporter: Hendrik Jan van Randen


We want to use Kafka for all the communication between our services. 

We have many high frequency topics (many messages per second), for which the Kafka mechanism of polling by consumers is efficient and suitable.

However, we also have low frequency topics (no or few messages per day). If such a low frequency message is produced, all consumers should consume it immediately. But it would be inefficient if all those consumers are polling continuously with a high frequency (e.g. each second), while no or few such messages are produced each day. Most efficient would be if for such a low frequency topic the consumers can register a callback (e.g. a REST endpoint) with Kafka, and Kafka calls that callback when a message is produced. Either supplying the message in the call (pro: consumer can handle it immediately), or just to notify the consumer that it should do a one time poll (pro: same message transfer mechanism as everywhere).

I just talked this over with Ryan Corrigan of Confluent.

In the coming weeks I will supply more information about the urgency of this issue.

I will be present at the Kafka Summit in London on May 16 & 17, 2023. Who of the Confluent technical staff, or Kafka committers, is present at the Kafka Summit to investigate possible solution directions for this?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)