[jira] [Assigned] (KAFKA-1409) oversized messages can slow down the brokers

["Guozhang Wang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Guozhang Wang reassigned KAFKA-1409:
------------------------------------

    Assignee: Guozhang Wang

> oversized messages can slow down the brokers
> --------------------------------------------
>
>                 Key: KAFKA-1409
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1409
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8.2
>            Reporter: Jun Rao
>            Assignee: Guozhang Wang
>            Priority: Critical
>             Fix For: 0.8.2
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are 2 main issues.
> 1. The broker first decompresses and then recompresses each message (to assign new offsets) before validating the message size (we have to do the validation after recompression since the message size could change). So, it can spend many secs to decompress/recompress an oversized message, only to be dropped later. While this was happening, a request thread was tied up, which reduced the capacity on the broker.
> 2. Both the fetch and producer requests need to hold a leader lock (per partition). So, if the producer is slow in appending the log, it will block other producer/fetch requests on the same partition.



--
This message was sent by Atlassian JIRA
(v6.2#6252)