You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by "Guozhang Wang (JIRA)" <ji...@apache.org> on 2016/05/11 21:42:12 UTC

[jira] [Created] (KAFKA-3704) Use default block size in KafkaProducer

Guozhang Wang created KAFKA-3704:
------------------------------------

             Summary: Use default block size in KafkaProducer
                 Key: KAFKA-3704
                 URL: https://issues.apache.org/jira/browse/KAFKA-3704
             Project: Kafka
          Issue Type: Bug
            Reporter: Guozhang Wang
            Assignee: Guozhang Wang
             Fix For: 0.10.0.0


As discovered in https://issues.apache.org/jira/browse/KAFKA-3565, the current default block size (1K) used in Snappy and GZIP may cause sub-optimal compression ratio for Snappy, and hence reduce throughput.

A better solution would be using the default block size, which is 32K in Snappy and 0.5K in GZIP. A notable side-effect is that with Snappy, this solution will require more extra memory allocated out side of the bufferpoll, by {{(32 - 1)K * num.total.partitions}} and hence users may need to bump up their JVM heap size, especially for MirrorMakers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)