[GitHub] [incubator-pinot] sunithabeeram opened a new issue #4007: Improve/Add completeness and freshness metrics for Pinot

[GitBox <gi...@apache.org>]

sunithabeeram opened a new issue #4007: Improve/Add completeness and freshness metrics for Pinot
URL: https://github.com/apache/incubator-pinot/issues/4007
 
 
   This issue tracks the need to expose better metrics in Pinot to expose completeness and freshness in the query path. Today, a lot of individual metrics are available for tracking, but its typically hard to say which queries were actually affected when issues do happen. Allowing tracking/metadata to be sent in query response helps with this. This issue tracks efforts being made for this. 
   NOTE: the tracking is on a best effort basis - we would like to track the most common possibilities (without tying ourselves to specific individual scenarios) without compromising the execution time of the query and continuing to keep the design/implementation as simple/intuitive as possible. 
   
   Details:
   ======
   Completeness:
   -------------------
   Currently, we set a "partial-response" metadata flag when a broker is unable to get a connection to the server or response from a server within configured timeout. We would like to expand this to cover:
   (a) cases where servers don't have the segments the broker is requesting.
   (b) broker's routing table is incomplete - for ex, when all the replicas of a given segment are ofline or in error state.
   
   Freshness:
   ---------------
   In the current scope, we limit the freshness of the data to realtime tables. We would like to measure how "recent" the data available for querying in realtime is. 
   
   More details for each of these will follow.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org