MirrorMaker producer partitioning

[Devon Crouse <de...@returnpath.com>]

Hello,

We're running 0.8.1.1, and having an issue with the MirrorMaker not evenly
distributing production over all topic partitions (
http://i.imgur.com/03u7bA9.png). Each producer seems to be using the
default behavior of sending to one partition for a period of time before
switching. Over time, the log size stays fairly even, but at any given
moment they all see very different message rates. There's no key on the
messages, but I've tried using a custom partitioner explicitly:

// com.returnpath.kafka.partitioners.RandomPartitioner
@Override
public int partition(Object key, int numPartitions) {
  return rand.nextInt() % numPartitions;
}

# Producer config
partitioner.class=com.returnpath.kafka.partitioners.RandomPartitioner

The mirror complained about the class until the jar was on the classpath,
so I know it's at least finding it, and there are no issues indicated in
the log. I tried returning 0 to force a single partition, also with no
resulting change in behavior.

Anyone else run into this, or have any ideas?


Thanks,
Devon