[jira] [Updated] (KAFKA-6127) Streams should never block infinitely

["John Roesler (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John Roesler updated KAFKA-6127:
--------------------------------
    Description: 
Streams uses three consumer APIs that can block infinite: {{commitSync()}}, {{committed()}}, and {{position()}}. Also {{KafkaProducer#send()}} can block. If EOS is enabled, {{KafkaProducer#initTransactions()}} also used to block (fixed in KAFKA-6446) and we should double check the code if we handle this case correctly.

If we block within one operation, the whole {{StreamThread}} would block, and the instance does not make any progress, becomes unresponsive (for example, {{KafkaStreams#close()}} suffers), and we also might drop out of the consumer group.

Thanks to [KIP-266|[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75974886],] the Consumer now has non-blocking variants that we can use, but the same is not true of Producer. We can add non-blocking variants to Producer as well, or set the appropriate config options to set the max timeout.

Of course, we'd also need to be sure the catch the appropriate timeout exceptions.

  was:
Streams uses three consumer APIs that can block infinite: {{commitSync()}}, {{committed()}}, and {{position()}}. Also {{KafkaProducer#send()}} can block. If EOS is enabled, {{KafkaProducer#initTransactions()}} also used to block (fixed in KAFKA-6446) and we should double check the code if we handle this case correctly.

If we block within one operation, the whole {{StreamThread}} would block, and the instance does not make any progress, becomes unresponsive (for example, {{KafkaStreams#close()}} suffers), and we also might drop out of the consumer group.

We might consider to use {{wakeup()}} calls to unblock those operations to keep {{StreamThread}} in a responsive state.

Note: there are discussion to add timeout to those calls, and thus, we could get {{TimeoutExceptions}}. This would be easier to handle than using {{wakeup()}}. Thus, we should keep an eye on those discussions. 


> Streams should never block infinitely
> -------------------------------------
>
>                 Key: KAFKA-6127
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6127
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 1.0.0
>            Reporter: Matthias J. Sax
>            Priority: Major
>              Labels: exactly-once
>
> Streams uses three consumer APIs that can block infinite: {{commitSync()}}, {{committed()}}, and {{position()}}. Also {{KafkaProducer#send()}} can block. If EOS is enabled, {{KafkaProducer#initTransactions()}} also used to block (fixed in KAFKA-6446) and we should double check the code if we handle this case correctly.
> If we block within one operation, the whole {{StreamThread}} would block, and the instance does not make any progress, becomes unresponsive (for example, {{KafkaStreams#close()}} suffers), and we also might drop out of the consumer group.
> Thanks to [KIP-266|[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75974886],] the Consumer now has non-blocking variants that we can use, but the same is not true of Producer. We can add non-blocking variants to Producer as well, or set the appropriate config options to set the max timeout.
> Of course, we'd also need to be sure the catch the appropriate timeout exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)