Kafka consumer - end of topic detection

[Vincenzo D'Amore <v....@gmail.com>]

Hi all,

I'm new to this mailing list, this is my first post, hello and thanks in
advance to everybody spend his/her time to read this.

I'm writing a Java application with kafka_2.11-0.10.1.0, I have just to
copy few data from a database to Solr.

In my application I've imagined to have two types of topics: control topic
and data topics. In particular there is only one control topic but there
can be many data topics.

So the Producer has 3 steps:

1. create a new topic X (where X is a GUID or a timestamp) and write all
data messages inside.
2. write a closure message that marks the end of data messages.
3. create a new message in the control topic declaring the creation of a
new topic X

On the other side there is a consumer that execute the following steps:

1. polls from the control data topics every few seconds
2. if see there is a new control message, starts to process the messages
data polling from the new topic X.
3. when the closure message is found the process stops and start again from
step 1

Now my idea is that: it is consumer responsibility see if a topic X is
complete, check if the producer has sent everything. I suppose just to
verify there is a closure message at end of topic.

I'm curious to know your suggestion, if check for a special closure message
at end of topic is enough. Or if there are already best practices useful to
handle this type of scenario.

Best regards and again thanks in advance for your time,
Vincenzo

-- 
Vincenzo D'Amore
email: v.damore@gmail.com
skype: free.dev
mobile: +39 349 8513251