You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "itschrispeck (via GitHub)" <gi...@apache.org> on 2023/06/26 19:57:44 UTC
[GitHub] [pinot] itschrispeck opened a new issue, #10976: _luceneRealtimeReaders is constantly increasing, causing text index creation lag
itschrispeck opened a new issue, #10976:
URL: https://github.com/apache/pinot/issues/10976
I added some logs to track [`_luceneRealtimeReaders`](https://github.com/apache/pinot/blob/release-0.12.1/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneIndexReaderRefreshThread.java#L49) size and it looks like this queue is constantly increasing as new readers are added when segments are created. As segments are destroyed the reader is marked to note the segment is destroyed, but it remains in the queue.
With ~30000 segments/week/server created we expect an increase of ~5 mins of index lag/week of server uptime (30k * (10ms sleep between reader refeshes + 0 ms refresh time)).
`RealtimeLuceneIndexReaderRefreshThread` loops through this queue, and it seems like we could check whether the segment is destroyed before readding the reader to the queue.
cc: @chenboat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #10976: _luceneRealtimeReaders is constantly increasing, causing text index creation lag
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #10976: _luceneRealtimeReaders is constantly increasing, causing text index creation lag
URL: https://github.com/apache/pinot/issues/10976
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org