Design for Kafka

[22...@guerrillamail.com]

Hi, I’m new. We are considering using Kafka for our next generation product. We have 300K devices that are connected to our cloud backend. We are currently using a set of web services for our headless client devices to send state information.  We are moving to a message-based interface. 

Does Kafka support a use case of many thousands of client/producers?  I can’t seem to find anyone using Kafka in this way. It appears most user have the small number of producers inside the data center.

It also seems the producers need to know a lot of state information about the Kafka cluster. Why should a client have to be involved selecting the right partition for a given topic? 

Or maybe I am total confused on how to correctly design for Kafka.

Any help / guidance is appreciated.





----
Sent using GuerrillMail.com
Block or report abuse: https://www.guerrillamail.com/abuse/?a=VFJxFx4gSLUTgw%2F68W4ecRzCA8WC1Q%3D%3D

Re: Design for Kafka

[Daniel Compton <de...@danielcompton.net>]

Hi there

One architecture pattern could be to have a number of app servers to receive the messages from the devices and send them to the Kafka cluster. This would let you handle authentication and authorisation from these devices, I don’t know your exact scenario but I don’t think you’d want 300K devices having direct write access to your Kafka cluster.

A second issue which would be more problematic is (as I understand it) every time a producer connects or disconnects to a cluster, it causes a rebalance of the producer -> partition mapping. Transient disconnections or restarts would cause some issues with that number of devices connecting and disconnecting.  

Daniel.


On Tuesday, 24 June 2014 at 12:07 pm, 229hgcqi8+ag6x9a7o@guerrillamail.com wrote:

> Hi, I’m new. We are considering using Kafka for our next generation product. We have 300K devices that are connected to our cloud backend. We are currently using a set of web services for our headless client devices to send state information. We are moving to a message-based interface.  
>  
> Does Kafka support a use case of many thousands of client/producers? I can’t seem to find anyone using Kafka in this way. It appears most user have the small number of producers inside the data center.
>  
> It also seems the producers need to know a lot of state information about the Kafka cluster. Why should a client have to be involved selecting the right partition for a given topic?  
>  
> Or maybe I am total confused on how to correctly design for Kafka.
>  
> Any help / guidance is appreciated.
>  
>  
>  
>  
>  
> ----
> Sent using GuerrillMail.com (http://GuerrillMail.com)
> Block or report abuse: https://www.guerrillamail.com/abuse/?a=VFJxFx4gSLUTgw%2F68W4ecRzCA8WC1Q%3D%3D
>  
>