You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "dang-stripe (via GitHub)" <gi...@apache.org> on 2024/02/22 22:35:47 UTC
[I] Expose metadata for table freshness through Pinot's query response [pinot]
dang-stripe opened a new issue, #12477:
URL: https://github.com/apache/pinot/issues/12477
We want to be able to tell how fresh the overall realtime table is as part of Pinot's query response.
`minConsumingFreshnessTimeMs` exists on the query response today but seems to have some limitations:
* If consuming segments are pruned or not queried, the timestamp may not represent the overall freshness of the table
* If the upstream topic data is sparse, the metric may not reflect how caught up the table actually is
cc @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Expose metadata for table freshness through Pinot's query response [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12477:
URL: https://github.com/apache/pinot/issues/12477#issuecomment-1960550089
I can see multiple potential solutions:
- Server maintaining freshness status, and put that in the query response when it is queried
- Limitation: We can only gather the freshness status for servers queried. If other server is lagging, this information won't be gathered
- Broker periodically ask servers for the freshness status, and returns it along with the queries
- Limitation: Freshness status won't be absolute accurate because it is fetched periodically
- Provide a Rest API on controller to gather freshness status from servers
- Limitation: Freshness status won't be returned along with the query response. User needs to call the Rest API in order to get it.
Let me know which solution fits your requirement the best
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org