[jira] [Commented] (KAFKA-1409) oversized messages can slow down the brokers

["Jun Rao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13975717#comment-13975717 ] 

Jun Rao commented on KAFKA-1409:
--------------------------------

The fix would be:
1. add an extra verification of the message size before decompression;
2. change the leader lock to be a read/write lock (only the leader updater will hold the write lock) so that slow producer requests don't block the fetch requests.
3. the byte-in rate includes over-sized messages, which is not consistent with message-in rate.

> oversized messages can slow down the brokers
> --------------------------------------------
>
>                 Key: KAFKA-1409
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1409
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8.2
>            Reporter: Jun Rao
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to assign new offsets) before validating the message size (we have to do the validation after recompression since the message size could change). So, it can spend many secs to decompress/recompress an oversized message, only to be dropped later. While this was happening, a request thread was tied up, which reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per partition). So, if the producer is slow in appending the log, it will block other producer/fetch requests on the same partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)