[jira] [Closed] (KAFKA-233) The producer's load balancing logic can send requests to dead brokers, when using the async producer option

["Neha Narkhede (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neha Narkhede closed KAFKA-233.
-------------------------------


> The producer's load balancing logic can send requests to dead brokers, when using the async producer option
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-233
>                 URL: https://issues.apache.org/jira/browse/KAFKA-233
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Neha Narkhede
>              Labels: newbie
>             Fix For: 0.8.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> The ZK producer, when used with the async producer option does the following 
> 1. Create a pool of async producers, one each for a broker registered under /broker/ids
> 2. On each send request, apply the Partitioner, to decide the broker and partition to send the data
> 3. Use the Async producer's send API to enqueue that data into the async producer's queue
> 4. When the data is dequeued by the ProducerSendThread, use the underlying sync producer to send it to the broker
> The load balancing decision is taken in step 2, before entering the queue. This leaves a window of error, equal to the queue length, when a broker can go down. When this happens, potentially, a queue worth of data can fail to reach a broker, and will be dropped by the EventHandler. 
> To correct this, the Producer, with the async option, needs to be refactored to allow only a single queue to hold all requests. And the application of the Partitioner should be moved to the end of the queue, in the EventHandler.



--
This message was sent by Atlassian JIRA
(v6.2#6252)