[GitHub] [pinot] navina commented on a diff in pull request #9621: Add record availability lag for Kafka connector

[GitBox <gi...@apache.org>]

navina commented on code in PR #9621:
URL: https://github.com/apache/pinot/pull/9621#discussion_r1000144296


##########
pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionLagState.java:
##########
@@ -27,12 +27,20 @@ public class PartitionLagState {
   protected final static String NOT_CALCULATED = "NOT_CALCULATED";
 
   /**
-   * Defines how far away the current record's offset / pointer is from upstream latest record
+   * Defines how far behind the current record's offset / pointer is from upstream latest record
    * The distance is based on actual record count.
    */
   public String getRecordsLag() {
     return NOT_CALCULATED;
   }
 
-  // TODO: Define record availability lag ($latest_record_consumption_time - $latest_record_ingestion_time)
+  /**
+   * Defines how soon after record ingestion was the record consumed by Pinot. That is, the difference between the
+   * time the record was consumed and the time at which the record was ingested upstream.
+   *
+   * @return Lag value in milliseconds
+   */
+  public String getRecordAvailabilityLag() {

Review Comment:
   I didn't want to call it "offset" lag as that term doesn't makes sense for anything other than kafka. in fact, `recordsLag` can be computed only for Kafka. Hence, I am using the same term as kafka's metric name for lag.
   
   For every other system, we can only compute a time-based lag. and it indicates when the record becomes available for consumption. I was considering calling it `getRecordAvailabilityLagMs`, if that helps clarify the meaning? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org