[jira] [Created] (KAFKA-1497) Change producer load-balancing algorithm in MirrorMaker

["Ivan Kunz (JIRA)" <ji...@apache.org>]

Ivan Kunz created KAFKA-1497:
--------------------------------

             Summary: Change producer load-balancing algorithm in MirrorMaker
                 Key: KAFKA-1497
                 URL: https://issues.apache.org/jira/browse/KAFKA-1497
             Project: Kafka
          Issue Type: Improvement
          Components: core
    Affects Versions: 0.8.1.1
            Reporter: Ivan Kunz


Currently the MirrorMaker uses the following way of spreading the load into configured producers :
            val producerId = Utils.abs(java.util.Arrays.hashCode(msgAndMetadata.key)) % producers.size()

This way if the producer side of MM uses different than the default "partitioner.class" messages within the same partition can get re-ordered. Also hashCode does not produce the same results on different machines (verified by testing) so cannot be safely used for partitioning between distributed systems connected via MM (for us message order preservation within a partition is a critical feature).
It would be great if the code above is changed to utilize the configured "partitioner.class". 
Something along the lines of  :
At the initialization:
      mmpartitioner = Utils.createObject[Partitioner](config.partitionerClass, config.props)      
During the processing:
            val producerId = mmpartitioner.partition(msgAndMetadata.key,producers.size())

This way the messages consumed and produced by MM can remain in the same order.




--
This message was sent by Atlassian JIRA
(v6.2#6252)