[jira] [Commented] (KAFKA-1755) Improve error handling in log cleaner

["Guozhang Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234780#comment-14234780 ] 

Guozhang Wang commented on KAFKA-1755:
--------------------------------------

Here are my two cents:

1. At the end of the day, Kafka will have two types of topics, one type only accepts keyed messages and log compaction is used; the other one accepts any message and log cleaning is used. Those two types of topics never exchange, i.e. once a topic is created with one of the two types, it will never change its type until deletion.

2. Compressed message will be supported with log compaction, which will de-serialize the message set and re-serialize.

3. With these two points in mind, I would suggest for now:
  a. Broker reject non-keyed messages for compacted topics.
  b. Broker reject compressed messages for compacted topics (this will be lifted after KAFKA-1374 is checked in).
  c. With this, it should never happen that compactor thread encountering a non-keyed / compressed (this will be lifted after KAFKA-1374); if it happens, this would be a FATAL error and we should throw an exception and halt the server. It indicates some operations are needed and there are some code fixes before it can be restarted.

> Improve error handling in log cleaner
> -------------------------------------
>
>                 Key: KAFKA-1755
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1755
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Joel Koshy
>            Assignee: Joel Koshy
>              Labels: newbie++
>             Fix For: 0.8.3
>
>
> The log cleaner is a critical process when using compacted topics.
> However, if there is any error in any topic (notably if a key is missing) then the cleaner exits and all other compacted topics will also be adversely affected - i.e., compaction stops across the board.
> This can be improved by just aborting compaction for a topic on any error and keep the thread from exiting.
> Another improvement would be to reject messages without keys that are sent to compacted topics although this is not enough by itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)