You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2014/08/20 23:45:26 UTC

[jira] [Created] (SAMZA-388) Log compaction on checkpoint topics fails with compression

Chris Riccomini created SAMZA-388:
-------------------------------------

             Summary: Log compaction on checkpoint topics fails with compression
                 Key: SAMZA-388
                 URL: https://issues.apache.org/jira/browse/SAMZA-388
             Project: Samza
          Issue Type: Bug
          Components: kafka
    Affects Versions: 0.8.0
            Reporter: Chris Riccomini


I have a job that has 10,000+ partitions that it's consuming from. After SAMZA-123, it's been switched to use the GroupBySystemStreamPartition strategy, which means it's got 10,000+ tasks, and thus, 10,000+ checkpoint messages being sent every minute.

To keep the checkpoint topic from getting too large, we enabled log compaction on the Kafka topic, but we discovered that the topic then grew to be very large. This behavior was triggered because we were sending compressed messages to the Kafka checkpoint topic.

Based on KAFKA-1374, it appears that we can't use compressed checkpoint topics with log compaction.

I'm mostly opening this ticket as a place holder for KAFKA-1374. Once the ticket is resolved, we can update the Samza code to default the checkpoint topics to be log compacted (with a small segment size), and not worry about the compression anymore.



--
This message was sent by Atlassian JIRA
(v6.2#6252)