[GitHub] [incubator-pinot] sunithabeeram opened a new pull request #4025: Extend stream interfaces to expose message metadata like ingestion timestamp

[GitBox <gi...@apache.org>]

sunithabeeram opened a new pull request #4025: Extend stream interfaces to expose message metadata like ingestion timestamp
URL: https://github.com/apache/incubator-pinot/pull/4025
 
 
   This PR provides the basic structure to expose freshness metrics as captured in https://github.com/apache/incubator-pinot/issues/4007. The stream interfaces are extended to provide an optional StreamMessageMetadata - this class currently exposes the ingestion timestamp of the message and can be extended to add other relevant metadata as needed. The ingestion timestamp is intended to reflect the time that the message was ingested into a streaming service such as Kafka and is used to track the freshness/lag of realtime message consumption in Pinot.
   
   *Backward compatibility considerations*
   The newly added methods to the interfaces are default methods and call the existing methods for backward compatibility. Stream extensions that would like to take advantage of freshness tracking, should override the default methods. However, just upgrading to the new version will not result in compilation failures.
   
   *Testing done*
   None. Once this change is reviewed, I will be updating https://github.com/apache/incubator-pinot/pull/3979 to expose both ingestion and indexed timestamps in a meaningful way and update the tests there.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org