[jira] [Updated] (SAMZA-839) KafkaSystemProducer should use the same partitioning hash function as Kafka's producer

["Dong Lin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dong Lin updated SAMZA-839:
---------------------------
    Description: 
Samza's KafkaSystemProducer class generates the partition key using:

{{abs(envelope.getPartitionKey.hashCode()) % numPartitions}}

However, Kafka's producer generates the partition key this way:

{{Utils.abs(Utils.murmur2(record.key())) % numPartitions}}

This makes it difficult for me to join 2 data sources on a common key when one source is produced by Samza and the other by a default Kafka producer.

As a work-around, I have to modify the upstream job (that uses the default kafka producer) to write with an explicit partition key using Samza's hashing logic. 

See https://docs.google.com/document/d/1iNSNQLDqCIfRiKz1MU3o6UMT_HxjgkgxpLhVcND4dYY/edit?usp=sharing for design.

  was:
Samza's KafkaSystemProducer class generates the partition key using:

{{abs(envelope.getPartitionKey.hashCode()) % numPartitions}}

However, Kafka's producer generates the partition key this way:

{{Utils.abs(Utils.murmur2(record.key())) % numPartitions}}

This makes it difficult for me to join 2 data sources on a common key when one source is produced by Samza and the other by a default Kafka producer.

As a work-around, I have to modify the upstream job (that uses the default kafka producer) to write with an explicit partition key using Samza's hashing logic. 


> KafkaSystemProducer should use the same partitioning hash function as Kafka's producer
> --------------------------------------------------------------------------------------
>
>                 Key: SAMZA-839
>                 URL: https://issues.apache.org/jira/browse/SAMZA-839
>             Project: Samza
>          Issue Type: Bug
>          Components: kafka
>    Affects Versions: 0.9.1
>            Reporter: Kishore Nallan
>            Assignee: Dong Lin
>            Priority: Major
>
> Samza's KafkaSystemProducer class generates the partition key using:
> {{abs(envelope.getPartitionKey.hashCode()) % numPartitions}}
> However, Kafka's producer generates the partition key this way:
> {{Utils.abs(Utils.murmur2(record.key())) % numPartitions}}
> This makes it difficult for me to join 2 data sources on a common key when one source is produced by Samza and the other by a default Kafka producer.
> As a work-around, I have to modify the upstream job (that uses the default kafka producer) to write with an explicit partition key using Samza's hashing logic. 
> See https://docs.google.com/document/d/1iNSNQLDqCIfRiKz1MU3o6UMT_HxjgkgxpLhVcND4dYY/edit?usp=sharing for design.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)