max.block.ms and TimeoutException on startup question

[Shane Gannon <sg...@gmail.com>]

Hi

I've a question I'm hoping the Kafka community can help me puzzle out. In
our setup we've configured *max.block.ms <http://max.block.ms>* to be 100
ms. For normal

    KafkaProducer.send(record, exceptionHandler)

calls this works fine. But when our app is starting up it'll typically take
~500 - 600 ms for the topic's metadata to be retrieved. As a result the
first 5 - 6 sends drop. i.e. After we ignore a TimeoutException.

I've been trying to figure out how to improve this situation. I've noticed
that the default *max.block.ms <http://max.block.ms>* timeout is 60,000 ms
(aka 1 minute). Defaulting to that value would likely sort out the initial
setup problem. But....

If the Kafka server is unreachable then send(..) waits until *max.block.ms
<http://max.block.ms>* expires. i.e. There is no HostUnreachable exception.
Hence a large timeout would introduce delays if Kafka was down.


*max.block.ms <http://max.block.ms>* is also used when the Kafka Buffer is
full. i.e. Additional calls block for *max.block.ms <http://max.block.ms>*.
If we have a large value that'll lead to performance issues when the system
is under stress.

Plus, TBH, if Kafka send is taking 55 seconds, for example, that should
also be treated as an error.

I was thinking about how we could more smartly address this. As we know, to
a large extent, our topics before hand could we use a method like

    public List<PartitionInfo> partitionsFor(String topic)

To retrieve the metadata when creating a new KafkaProducer? i.e. Is this a
common & viable solution?

What other options does the Kafka community employ?

Regards
Shane