Enabling Top-k extraction module to Kafka

[Janagan Sivagnanasundaram <ja...@gmail.com>]

Hi,

Is there any libraries or any other way to implement top-k quality
extraction algorithm (top-k info per window/batch) to the existing Kafka,
which enable the consumer to get the top-k quality information once the
consumer subscribed to a particular topic.

Scenario: Producer free to produce message and once the subscriber
subscribed to a particular topic, subscriber will be able to get top k
quality information per batch/window as Kafka processed/

Thanks.

Re: Enabling Top-k extraction module to Kafka

[Eric Lalonde <er...@autonomic.ai>]

> On Nov 21, 2016, at 6:41 PM, Janagan Sivagnanasundaram <ja...@gmail.com> wrote:
> 
> Hi,
> 
> Is there any libraries or any other way to implement top-k quality
> extraction algorithm (top-k info per window/batch) to the existing Kafka,
> which enable the consumer to get the top-k quality information once the
> consumer subscribed to a particular topic.
> 
> Scenario: Producer free to produce message and once the subscriber
> subscribed to a particular topic, subscriber will be able to get top k
> quality information per batch/window as Kafka processed/
> 
> Thanks.

The top songs by genre example in the confluentinc/examples github repo solves the problem of ‘top 5 songs by genre’.

https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java

I am not suggesting it is a solution, but a good start.

- Eric