Kafka control messages

["Young, Ben" <Be...@fisglobal.com>]

Hi,

We have a situation where we are streaming data in a Kafka topic to be processed. The topic has multiple partitions. From time to time we need to add control messages to the stream and this needs to be coherent with the data steam (e.g. "replay from point X to this new location"). Because of the way Kafka partitions it seems hard to get these control messages to every consumer, no matter how they have been assigned, in the right place in the queues. The way I see it there are two options:

1) Have the producer put these control messages on every partition. We could get the number of partitions and manually post the same message on every partition, which will guarantee it gets read in the right place.
2) Reserve a special partition (e.g. 0) to be a control one, and have all consumers subscribe to that one. I don't think this gives the guarantees we want though.

The other option is to use a separate queue for these, but then we'd need some special logic to ensure they have their effect at the right point in the queue.

Has anyone else implemented anything like this? Am I barking up the wrong tree?

Thanks,
Ben

Ben Young
Principal Software Engineer - Risk
Risk & Compliance
T:  +44 (0) 1865 578140
E: ben.young@fisglobal.com
FIS | Empowering the Financial World
FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered Office: Level 39, 25 Canada Square, London E14 5LQ

The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

RE: Kafka control messages

[adrien ruffie <ad...@hotmail.fr>]

Hi Ben,


it's depend on your consumer group configuration.

If you have all consumer arein differents group (only one consumer for each consumer group), you can use 2) because all the consumer instances have different consumer group,

then the control records will be broadcasted to all your consumer processes.


But if you have several consumer in each consumer group, you must implement the 1) solution, because records will effectively be load balanced over the consumer processes.

So if only one consumer by group will read it...


best regards,


Adrien

________________________________
De : Young, Ben <Be...@fisglobal.com>
Envoyé : lundi 19 février 2018 13:53:05
À : users@kafka.apache.org
Objet : Kafka control messages

Hi,

We have a situation where we are streaming data in a Kafka topic to be processed. The topic has multiple partitions. From time to time we need to add control messages to the stream and this needs to be coherent with the data steam (e.g. "replay from point X to this new location"). Because of the way Kafka partitions it seems hard to get these control messages to every consumer, no matter how they have been assigned, in the right place in the queues. The way I see it there are two options:

1) Have the producer put these control messages on every partition. We could get the number of partitions and manually post the same message on every partition, which will guarantee it gets read in the right place.
2) Reserve a special partition (e.g. 0) to be a control one, and have all consumers subscribe to that one. I don't think this gives the guarantees we want though.

The other option is to use a separate queue for these, but then we'd need some special logic to ensure they have their effect at the right point in the queue.

Has anyone else implemented anything like this? Am I barking up the wrong tree?

Thanks,
Ben

Ben Young
Principal Software Engineer - Risk
Risk & Compliance
T:  +44 (0) 1865 578140
E: ben.young@fisglobal.com
FIS | Empowering the Financial World
FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered Office: Level 39, 25 Canada Square, London E14 5LQ

The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.