How to shard and replicate with python and kafka

[David Montgomery <da...@gmail.com>]

Hi,

I am using druid that consumes from kafka 8.  Using python-kafka I write to
kafka so 1 server and I have 2 partitions.

Given this setup,

1) How do I shard random? I assume a shard == partition...I hope.. and I
assune the client is responsible for sharding

For Example
rand = random.choice([0,1])
req = ProduceRequest(topic=test_topic,
partition=rand,messages=[create_message(test_payload)])
response = kafka.send_produce_request(payloads=[req], fail_on_error=True)

Is the above best practice?

If I extend to 2 servers with 2 partitions then what does sharding look
like?
In my kafka client:
kafka = KafkaClient('111.111.111.111:9092,222.222.222.222:9092')

So with 2 servers and 2  partitions each does kafka write only to one
partiton out of the 4 total across 2 servers?  Thus the client does a round
robin?


In terms of replication.....what I gather from the docs, this happens on
the broker node, not the python client.  Thus all topics have a rep factor
of 2 using the below.

e.g.

server.properties
# Replication configurations
num.replica.fetchers=2


Thanks