Questions about Apache Kafka messages type/size and publish/subscribe

[Heloise Chevalier <hc...@gmail.com>]

Hi,

I'm not entirely certain this is the right place to ask, but I have
questions about the functioning of Apache Kafka to implement a
publish/subscribe messaging system. I am investigating Kafka to see if it
fits the needs of the company I work for, and I have quite a few questions
I can't find any answer to either in the official doc or on the internet :

On the type and size of Kafka messages, I read the doc on RecordBatch and
Records, but I couldn't find anything about the type of data expected in
the Record value. From what I read on different websites and tutorials it
seems that data can be anything as long as it can be converted into byte
arrays. Moreover, I read that the default size of a message is 1MB, but as
it is possible to configure it, what would be the maximum size of a
message?

- Is it possible to send files? Basically what we would want to do is
streaming pictures, right now the system in use does not allow to send
files so they are sent as binary data and assembled later. From what I
understand of Kafka, files can be sent in a binary way, split and to be
reassembled after.

- Is it possible to send binary frames? Right now we have a few "custom"
binary frame and we would like to know if this would still be possible when
using Kafka.

- Is it possible to send simple messages, such as a string + an int or a
string + a string...?

- What is the max size of a queue, if there is one? We would like to know
what is the max volume of data that can be sent instantaneously (depending
on the size of the queue, I suppose).

On publish-subscribe more specifically :

- Is it possible for a consumer to subscribe to a group of publishers?
Otherwise I guess the easiest way to do so is to have all publishers from
that group publish their messages in a specific topic.

- Is it possible to subscribe to a group of topics? For example if we have
several topics starting by "data", like data_1, data_2 etc, would it be
possible to have consumers subscribe to "data*" ?

- Is it possible to have sub-topics, and even sub-sub-sopics? From what I
understand so far it doesn't seem possible, as topics are divided into
partitions for replication.



I know this is a lot of questions and that they are not really related. I
hope I made myself clear enough and I thank you in advance for your answers.

Regards,

Héloïse Chevalier