[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future

["Moses Nakamura (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342739#comment-17342739 ] 

Moses Nakamura commented on KAFKA-3539:
---------------------------------------

We've been wrestling with this problem for a while.  I think my company would like to start the process to try to patch it.  However, it's definitely thorny, so we had some issues we wanted to talk about!  The main questions seem to be:
 # Which thread should wait for the partition metadata?
 # How can we preserve the ordering semantics that send gives us today?

I think the ideal answer to (1) would be, "no thread" and that we should do it asynchronously.  This would be a pretty invasive change though, so I think it's probably more expedient to pass in a threadpool and do it there.  How would you feel about optionally passing in a threadpool to KafkaProducer#send, or else on construction to KafkaProducer?

For (2), it seems like we have two plausible approaches, either pass async data to the RecordAccumulator, or else layer another queue on top.  I think layering another queue on top would be less invasive, so I would lean toward that approach.  We could have a central queue that orders all of the work, conditioned on the metadata that we're waiting for being ready.

Does that sound good to you?  Should I put up a patch so we can talk about it?

> KafkaProducer.send() may block even though it returns the Future
> ----------------------------------------------------------------
>
>                 Key: KAFKA-3539
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3539
>             Project: Kafka
>          Issue Type: Bug
>          Components: producer 
>            Reporter: Oleg Zhurakousky
>            Priority: Critical
>              Labels: needs-discussion, needs-kip
>
> You can get more details from the users@kafka.apache.org by searching on the thread with the subject "KafkaProducer block on send".
> The bottom line is that method that returns Future must never block, since it essentially violates the Future contract as it was specifically designed to return immediately passing control back to the user to check for completion, cancel etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)