[jira] [Created] (SAMZA-918) Disable check to turn off compression for changelog topics

["Jagadish (JIRA)" <ji...@apache.org>]

Jagadish created SAMZA-918:
------------------------------

             Summary: Disable check to turn off compression for changelog topics
                 Key: SAMZA-918
                 URL: https://issues.apache.org/jira/browse/SAMZA-918
             Project: Samza
          Issue Type: Improvement
            Reporter: Jagadish
            Assignee: Jagadish


There was a Kafka bug that would cause the log cleaner thread on the broker to crash when it was compacting compressed topics. Since, changelog topics are log-compacted, we have enabled the override to explicitly disable compaction if a k-v store topic is compressed. This kafka bug https://issues.apache.org/jira/browse/KAFKA-1374 is fixed as of kafka 0.9.0.2. We must ensure that we disable that check as we move to kafka 0.9.0.2. 

Enabling compression has the following benefit:
1. When sending large messages (the limit of a kafka message is typically 1M), with compression and batching, the message sizes get reduced significantly. This helps Samza customers to write larger data to their K-V stores (because the size of the write you can do to a kafka-changelog backed store is dependent on the size of the Kafka message)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)