[jira] [Commented] (KAFKA-1586) support sticky partitioning in the new producer

["Jun Rao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092237#comment-14092237 ] 

Jun Rao commented on KAFKA-1586:
--------------------------------

One way to address this issue is introduce a new config "partition.sticky.time.ms" in the new producer. The producer will then stick to a partition for the configured amount of time before switching to another. "partition.sticky.time.ms" can default to 0, which means every message will switch to a new partition.

> support sticky partitioning in the new producer
> -----------------------------------------------
>
>                 Key: KAFKA-1586
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1586
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8.2
>            Reporter: Jun Rao
>
> If a message doesn't specify a key or a partition, the new producer selects a partition for each message in a round-robin way. As a result, in a window of linger.ms, messages are spread around in all partitions of a topic. Compared with another strategy that assigns all messages to a single partition in the same time window, this strategy may not compress the message set as well since the batch is smaller. Another potential problem with this strategy is that the compression ratio could be sensitive to the change of # partitions in a topic. If # partitions are increased in a topic, the produced data may not be compressed as well as before. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)