Re: How to use RPC mechanism in Kafka?

[Julien Moumné <ju...@quicksign.com>]

Several answers in this thread seem to indicate Kafka can not be used to 
simulate RPC.

In Joe Stein's Real-Time Distributed and Reactive Systems presentation, 
RPC style calls look like they can be part of a system built using Kafka.

Reference : https://youtu.be/wRzL-0VSpTY?t=5m19s

The video does not go into details on how to achieve the 
request/response paradigm using Kafka.

Here is a solution we came-up with, please comment if there is a better 
approach or if you see any shortcomings.

In the diagram from the video, the "Producer box" needs to :
  - accept incoming client requests
  - produce messages into a Kafka topic
  - keep a registry of pending client requests
  - consume "Response" messages on a different Kafka topic
  - respond to client requests by correlating messages issued from the 
"Response" topic

If the "Producer box" is replicated, response messages could end-up in 
the wrong "Producer box". This is due to consumer partitioning.

One way to solve this issue is to attribute a unique consumer group to 
each "Producer box". This way, each consumer will receive every 
"Response" messages.

This is clearly not optimized and we would welcome ideas to improve the 
solution.