Broker Health check

[Brice Dutheil <br...@gmail.com>]

Hi,

I and my team are beginning with kafka so I/we may not have read everything
or understood everything yet.

The context is we’d like to bootstrap kafka in our own codebase, and first
step is to have a deployable walking skeleton that the team can grow along
our experience and needs.

We are starting with kafka 0.9.0.0, and surprisingly there’s no status
command for the broker, so we decided to do a comprehensive check but we
are still a bit confused on how to write a healthcheck that is not fragile.

Currently we would like to produce a message (e.g. a timestamp) on some
healthcheck topic, and expect a consumer to read this message, a grep would
check that the message has the same value.
However we are facing some behaviour that we don’t yet quite understand.

# first emit the message
timestamp=$(date -u "%Y-%m-%dT%H:%M%SZ)"echo $timestamp |
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
healthcheck
# try to consume the same message
bin/kafka-console-consumer.sh --new-consumer --bootstrap-server
localhost:9092 --topic healthcheck --max-messages 1 --from-beginning |
grep $timestamp

[ 0 == $? ] && echo "kafka onfline" || echo "kafka offline"

In this case the consumer does not always *find* the matching message. We
have tried different options there ; with or without --max-messages,
--from-beginning, --timeout-ms, it does’nt behave as we thought by reading
the documentation. Sometime the message is read some time it isn’t, but we
don’t get why.
------------------------------

To be noted, I don’t understand how it is possible to read the latest
message,

> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic yat --partitions 1 --replication-factor 1
Created topic "yat".

> echo bob | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092

> echo charles | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092
> echo yohan | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092
> echo carlos | bin/kafka-console-producer.sh --topic yat --broker-list localhost:9092
# Running the simple command just outputs nothing, and I need to kill
the process with ctrl-c, note the messages count is off by 1, or I'm
missing something
> bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic yat
^CProcessed a total of 1 messages
# Same with `--from-beginning`, the pocess don't finish up, and prints
everything since the beginning (that is to be expected).
bin/kafka-console-consumer.sh --new-consumer --bootstrap-server
localhost:9092 --topic yat --from-beginning
bob
charles
yohan
carlos
carlos
yohan
brice
^CProcessed a total of 8 messages
# With `--from-beginning` and `--max-messages` the consumer stops at
the very first message
[root@3af0ccad3182 kafka]# bin/kafka-console-consumer.sh
--new-consumer --bootstrap-server localhost:9092 --topic yat
--from-beginning --max-messages 1
bob
Processed a total of 1 messages
# With `--max-messages` the latest message doesn't appear either and I
have to kill the consumer too
> bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic yat --max-messages 1
^CProcessed a total of 1 messages

Also note the same happen with the old consumer with --zookeeper option.

Is there some limitation with the console consumer regarding offsets ? Is
there something I’m missing.

Should we write our own healthcheck ?

Thanks in advance for any feedback, pointers, etc.

Cheers,
— Brice
​