File Transfers between two systems using Kafka

[Mohammed Manna <ma...@gmail.com>]

Hello,

I currently have the following understanding of KafkaProducer and
KafkaConsumer:

1) If I send a file, it's broken down in lines using some default delimiter
(LF or \n).
2) Therefore, if 2 producers publish 2 different files to the same topic,
that doesn't mean that they are going as two different files as a whole.
Instead, every line will be appended to the log partitions and a consumer
will read it from the log partitions in the order of arrival.

Assuming that the following is correct, I believe simple Producer Consumer
usage to transfer files is not the correct approach (Probably connector API
is the solution here). Since Kafka Website says that "Log Aggregation" is a
very popular use case, I was wonder if someone has any example projects or
website which demonstrates file exchange examples using Kafka.

KR,