[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics

["Joel Koshy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037669#comment-14037669 ] 

Joel Koshy commented on KAFKA-1374:
-----------------------------------

I had started on this a while ago, but did not finish. Here's a WIP patch
that doesn't quite work yet:
https://gist.github.com/jjkoshy/4657a44e52e3f88be1c1

Another nuance with compression and compaction is what compression-codec do
we use when writing out the compacted data?

We could adopt a broker-side compression config that can be overridden on a
per-topic basis. This would not only enable a consistent compression codec
for each topic, but it will also make the above decision more
straightforward - i.e., write out compacted messages in the configured
compression codec for that topic (or broker-default if the topic does not
have any override).

Will file a separate jira for the above.



> LogCleaner (compaction) does not support compressed topics
> ----------------------------------------------------------
>
>                 Key: KAFKA-1374
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1374
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Joel Koshy
>             Fix For: 0.8.2
>
>
> This is a known issue, but opening a ticket to track.
> If you try to compact a topic that has compressed messages you will run into
> various exceptions - typically because during iteration we advance the
> position based on the decompressed size of the message. I have a bunch of
> stack traces, but it should be straightforward to reproduce.



--
This message was sent by Atlassian JIRA
(v6.2#6252)