Mirrormaker only publishing to a single partition on destination cluster

[Stephen Powis <sp...@salesforce.com>]

Hey!

I'm using kafka 0.9.0.1 and trying to replicate a cluster from one
datacenter to another.  mirror-maker properly connects to my source cluster
and consumes messages, but for some reason is only publishing to a single
partition for my topic in the destination cluster.  So all of my partitions
for the topic are empty, except one, which contains everything from the
source cluster.

Has anyone seen this behavior before?  I must have something misconfigured,
but am unable to figure it out from reviewing the online docs.

Thanks!
Stephen

Re: Mirrormaker only publishing to a single partition on destination cluster

[Stephen Powis <sp...@salesforce.com>]

Took me a bit but worked out my issue.  Appears like all of the messages in
my topics were being hashed to a single partition using the
DefaultPartitioner.  Ended up rolling a simple round robin partitioner and
contributed that back via this ticket:
https://issues.apache.org/jira/browse/KAFKA-3333

Thanks
Stephen

On Thu, Mar 3, 2016 at 11:33 AM, Stephen Powis <sp...@salesforce.com>
wrote:

> Hey!
>
> I'm using kafka 0.9.0.1 and trying to replicate a cluster from one
> datacenter to another.  mirror-maker properly connects to my source cluster
> and consumes messages, but for some reason is only publishing to a single
> partition for my topic in the destination cluster.  So all of my partitions
> for the topic are empty, except one, which contains everything from the
> source cluster.
>
> Has anyone seen this behavior before?  I must have something
> misconfigured, but am unable to figure it out from reviewing the online
> docs.
>
> Thanks!
> Stephen
>