[jira] [Resolved] (KAFKA-12980) Allow consumers to return from poll when position advances due to aborted transactions

["Jason Gustafson (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-12980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Gustafson resolved KAFKA-12980.
-------------------------------------
    Fix Version/s: 3.2.0
       Resolution: Fixed

> Allow consumers to return from poll when position advances due to aborted transactions
> --------------------------------------------------------------------------------------
>
>                 Key: KAFKA-12980
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12980
>             Project: Kafka
>          Issue Type: Improvement
>          Components: consumer
>            Reporter: Chris Egerton
>            Assignee: Chris Egerton
>            Priority: Major
>             Fix For: 3.2.0
>
>
> When {{Consumer::poll}} is invoked on a topic with an open transaction, and then that transaction is aborted, {{poll}} does not return unless there are other records available in that topic after the aborted transaction.
> Instead, {{poll}} could return in this case, even when no records are available.
> This facilitates reads to the end of a topic where the end offsets of a topic are listed and then a consumer for that topic is polled until its [position|https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#position(org.apache.kafka.common.TopicPartition)] is at or beyond each of those offsets (for example, [Connect does this|https://github.com/apache/kafka/blob/fce771579c3e20f20949c4c7e0a5e3a16c57c7f0/connect/runtime/src/main/java/org/apache/kafka/connect/util/KafkaBasedLog.java#L322-L345] when reading to the end of any of its internal topics).
> We could update the existing language in the docs for {{Consumer::poll}} from
> {quote}This method returns immediately if there are records available.
> {quote}
> to
> {quote}This method returns immediately if there are records available or if the position advances past control records.
> {quote}
>  
> A workaround for existing users who would like to see this is to use short poll intervals and manually check the consumer's position in between each poll, but this is fairly tedious and may lead to excess CPU and network utilization depending on the latency requirements for knowing when the end of the topic has been reached.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)