Kafka use case query

[lrnop <lr...@protonmail.com>]

Hello,



We have a number of different scenarios within our company that we are considering Kafka for.



There is one case in particular that has caused debate. The relevant characteristics are:




-
Very high throughput - 1000's of messages/second.

-
Very bursty traffic with unpredictable surges in requests from external sources.

-
Management of long running request/response sessions.

-
In Kafka terminology the producer that issues a request must also be the consumer for the response.

-
Very short message validity - If a response is not received within a few seconds then the request is invalid.

-
The consumers of the requests need per-request interaction with a database, introducing further sources of latency and unpredictable processing rate.


Not everyone is convinced Kafka is a good choice for this particular case. There are others - such as the backbone for post-processing of TB of transaction records - where everyone agrees Kafka would be useful.



I'd be very grateful for any feedback.



Thanks!



HJ











Sent from [ProtonMail](https://protonmail.ch), encrypted email based in Switzerland.