Priority queues using kafka topics and a consumer group?

[Richard Ney <ri...@lookout.com.INVALID>]

Trying to see if there is a way to implement priority topics where a single
consumer group is reading from multiple topics each denoting a level of
priority and having the Kafka client automatically prioritize reads against
the message in the higher priority topics.

Re: Priority queues using kafka topics and a consumer group?

[Ricardo Ferreira <ri...@riferrei.com>]

Richard,

There is; but it doesn't look like what you would do in a typical 
messaging technology. As you may know Apache Kafka® is not messaging but 
a distributed streaming platform based on the concept of commit log. The 
main characteristic of a commit log is that they keep records in the 
same order that they were written (hence the immutable nature) and 
therefore the ordering cannot be changed in a broker level. With this in 
mind what you can do is:

A. *Ensure Priority using Partitions*: Break down the topic into 
multiple partitions where each partition represents a priority. Change 
your producers to use a custom partitioner that writes records into a 
given partition depending on the priority logic such as: "if the order 
is from North America then send to partition 0 that represents platinum 
orders.". This way your consumer groups will consume them as they come 
based on their natural order without worrying about handling priority 
themselves.

B. *Using the Resequencer Pattern 
<https://www.enterpriseintegrationpatterns.com/patterns/messaging/Resequencer.html>*: 
Have a dedicated stream processor (implemented as a Kafka consumer, as a 
Kafka Streams app, or as a ksqlDB app) that reads all the records from 
the input topic and perform a routing to specific topics based on the 
data contained in the records -- preferably data from the headers for 
improved performance. This way you can have dedicated consumer groups 
each one reading from specific topics without worrying about handling 
priority themselves.

Either option works, though personally I fancy option A since it 
provides a less intrusive solution with less layers being added to the 
final architecture.

-- Ricardo

On 7/21/20 12:52 PM, Richard Ney wrote:
> Trying to see if there is a way to implement priority topics where a single
> consumer group is reading from multiple topics each denoting a level of
> priority and having the Kafka client automatically prioritize reads against
> the message in the higher priority topics.
>