[jira] [Commented] (KAFKA-233) The producer's load balancing logic can send requests to dead brokers, when using the async producer option

["Ashwanth Fernando (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696204#comment-13696204 ] 

Ashwanth Fernando commented on KAFKA-233:
-----------------------------------------

Hi, I think I am seeing this already implemented in 0.8. Can we close this if that is the case?
                
> The producer's load balancing logic can send requests to dead brokers, when using the async producer option
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-233
>                 URL: https://issues.apache.org/jira/browse/KAFKA-233
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.7
>            Reporter: Neha Narkhede
>              Labels: newbie
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> The ZK producer, when used with the async producer option does the following 
> 1. Create a pool of async producers, one each for a broker registered under /broker/ids
> 2. On each send request, apply the Partitioner, to decide the broker and partition to send the data
> 3. Use the Async producer's send API to enqueue that data into the async producer's queue
> 4. When the data is dequeued by the ProducerSendThread, use the underlying sync producer to send it to the broker
> The load balancing decision is taken in step 2, before entering the queue. This leaves a window of error, equal to the queue length, when a broker can go down. When this happens, potentially, a queue worth of data can fail to reach a broker, and will be dropped by the EventHandler. 
> To correct this, the Producer, with the async option, needs to be refactored to allow only a single queue to hold all requests. And the application of the Partitioner should be moved to the end of the queue, in the EventHandler.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira