[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

["Jay Kreps (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388879#comment-14388879 ] 

Jay Kreps commented on KAFKA-2076:
----------------------------------

This makes sense, and would be useful.

The question is whether we want to do this now or later. We have talked about generalizing the existing offset request and exposing that (e.g. to allow also querying for the first offset, and getting offsets by time). If we are going to do that we almost need to think through that full api, as this will be closely related. However I am not very happy with either the protocol that that uses (the request is super hacky) and the way we assign times is quite buggy (since it is based on server time not event time), so I am not wild about exposing it until we have a better story.

Two corner cases to consider the behavior for:
1. I ask for the log end offset for a partition I am not subscribed to
2. I ask for the log end offset for a partition I am subscribed to but for which a fetch request has not yet been issued

> Add an API to new consumer to allow user get high watermark of partitions.
> --------------------------------------------------------------------------
>
>                 Key: KAFKA-2076
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2076
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Jiangjie Qin
>
> We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)