building kafka minimal kafka cluster that tolerates a loss of one node.

[Vitaliy Semochkin <vi...@gmail.com>]

Hi

What should be parameters for a broker, topic and producer
if I want to build a cluster of 3+ kafka brokers
which will survive occasional one node crashes (including rack crash)
and guaranties with at least once delivery.


If numbers of brokers>=3 my own thoughts are:
broker.rack = each of 3 brokers should have different physical rack
broker/topic replication-factor=3
broker/topic min.insync.replicas=2
producer acks=all(-1)

Questions:
Will this configuration solve the task?
Which parameters are responsible for the time when the cluster detects
that leader is down?
How long(approximately) will it take for a cluster to elect the new
leader for partitions of a broken broker?
Will the message that was sent to an old partition leader be resend to
a new leader transparently, or the API will throw an exception and
producer should re-send the message?
Is it possible to avoid message re-ordering?
Is it possible to avoid message loss if node goes down?
Is it possible to avoid split brain in case I have only 3 brokers and
min.insync.replicas=2?
Is it possible to avoid message loss if I have more brokers (e.g. 15)
and min.insync.replicas=2?


PS Any links to articles/docs/bluepreints are welcome.

Best regards,
Vitaliy