[jira] [Commented] (KAFKA-10009) Add method for getting last record offset in kafka partition

["Werner Daehn (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-10009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173661#comment-17173661 ] 

Werner Daehn commented on KAFKA-10009:
--------------------------------------

It is actually worse than that.

There could be log compaction or other reasons why the offset value is not a dense number set.

Without such functionality, how do you reliably know that you have read all data from the topic?
 * Option 1: execute a poll(1second) and if it returns no data, that means there is no more data. But maybe the network was busy, so 1 second is not enough. 10 seconds? 1 minute, 10 minutes? I don't want to wait for ten minutes just to decrease the probability there is more data and I just have not received it yet.
 * Option 2: First call the endOffset(), then you know the high water mark is offset=100. So you poll until you received the record with offset=99 and then you know you have gotten the last record. But what if there is no record with offset=99? Again, you will wait forever.

> Add method for getting last record offset in kafka partition
> ------------------------------------------------------------
>
>                 Key: KAFKA-10009
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10009
>             Project: Kafka
>          Issue Type: New Feature
>          Components: clients, consumer
>            Reporter: Yuriy Badalyantc
>            Priority: Minor
>
> As far as I understand, at the current moment, there is no reliable way for getting offset of the last record in the partition using java client. There is {{endOffsets}} method in the consumer. And usually {{endOffsets - 1}} works fine. But in the case of transactional producer, topic may contain offsets without a record. And {{endOffsets - 1}} will point to the offset without record.
> This feature will help in situations when consumer application wants to consume the whole topic. Checking of beginning and last record offset will give lower and upper bounds for consuming. Of course, it is doable with the current consumer implementation, but I need to check {{position}} after each poll.
> Also, I believe that this feature may help with monitoring and operations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)