You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "sajjad-moradi (via GitHub)" <gi...@apache.org> on 2023/11/03 16:53:06 UTC
[I] Enhance Force-Commit Endpoint to Stagger Segment Commit for Different Partitions [pinot]
sajjad-moradi opened a new issue, #11950:
URL: https://github.com/apache/pinot/issues/11950
Currently, when calling the force commit endpoint, it triggers the simultaneous segment commit for all consuming segments. However, this approach poses two significant issues:
- With a large number of partitions, the table-wide consumption lag could be substantial during the segment commit process, which can last for minutes.
- Segment commit, being memory-intensive, can lead to multiple GC pauses and degrade the performance of query execution, when multiple segment commits occur on the same server.
To address these concerns, we can extend the force-commit endpoint to include a parameter that specifies a delay between the commit processes of different partitions. This modification would distribute the segment commit over a more extended period, potentially alleviating the mentioned problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Enhance Force-Commit Endpoint to Stagger Segment Commit for Different Partitions [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11950:
URL: https://github.com/apache/pinot/issues/11950#issuecomment-1797129836
When force commit is triggered, I believe server side should have a semaphore controlling the concurrent segment commit allowed.
We do detect a hotspot on controller side, which can be fixed with #11943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org