[GitHub] [pinot] elonazoulay commented on pull request #9490: Eliminate explosion of metrics due to gapfill queries

[GitBox <gi...@apache.org>]

elonazoulay commented on PR #9490:
URL: https://github.com/apache/pinot/pull/9490#issuecomment-1262472781

   This should be labelled with bugfix. 
   Context: 
   We observed that the jmx prometheus scraper timed out when scraping metrics after about 15 minutes of a broker starting. 
   This was due to the metrics response being ~99mb eventually never returning a response when prometheus scrapes metrics.
   99% of those metrics were due to table names being set to gap fill queries.
   Once we deployed this change the metrics went back to normal (< 1mb response size) and prometheus scraper is once again able to scrape broker metrics.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org