Topic per entity

[Alex Buchanan <bu...@gmail.com>]

Hey Kafka community.

I'm researching possible architecture for a distributed data processing
system. In this system, there's a close relationship between a specific
dataset and the processing code. The user might upload a few datasets and
write code to run analysis on that data. In other words, frequently the
analysis code pulls data from a specific entity.

Kafka is attractive for lots of reasons:
- I'll need messaging anyway
- I want a model for immutability of data (mutable state and potential job
failure don't mix)
- cross-language clients
- the change stream concept could have some nice uses (such as updating
visualizations without rebuilding)
- Samza's model of state management is a simple way to think of external
data without worrying too-much about network-based RPC
- as a source of truth data store, it's really simple. No mutability,
complex queries, etc. Just a log. To me, that helps prevent abuse and
mistakes.
- it fits well with the concept of pipes, frequently found in data analysis

But most of the Kafka examples are about processing a large stream of a
specific _type_, not so much about processing specific entities. And I
understand there are limits to topics (file/node limits on the filesystem
and in zookeeper) and it's discouraged to model topics based on
characteristics of data. In this system, it feels more natural to have a
topic per entity so the processing code can connect directly to the data it
wants.

So I need a little guidance from smart people. Am I lost in the rabbit
hole? Maybe I'm trying to force Kafka into this territory it's not suited
for. Have I been reading too many (awesome) articles about the role of the
log and streaming in distributed computing? Or am I on the right track and
I just need to put in some work to jump the hurdles (such as topic storage
and coordination)?

It sounds like Cassandra might be another good option, but I don't know
much about it yet.

Thanks guys!