You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "tarun11Mavani (via GitHub)" <gi...@apache.org> on 2025/08/19 12:44:26 UTC
[I] Handle segment flush threshold for commit time compaction enabled tables [pinot]
tarun11Mavani opened a new issue, #16632:
URL: https://github.com/apache/pinot/issues/16632
In #16344, we introduced a new feature to enable commit time compaction for upsert table. It removes the invalid records before committing the segment which means pre commit and post commit row count and segment size are different. Currently, all the segment flush threshold operates on the assumption that we commit all rows that we consume.
After this change, we have seen that we start committing segment much earlier even though we have added initial logic to estimate the pre-commit row for next segment based on pre-commit and post-commit row count and post-commit segment size of the old segment. The current logic is not robust to handle different compaction ratio (amount of repeat records for the same PK in a segment) and due to which size based threshold commits segment much faster.
We should introduce a new config called `realtime.segment.flush.threshold.precommit.segment.size` to allow user to set limit for pre-commit segment size instead of limiting them to use the current version which is specifically for post commit segment size.
Note: This is a placeholder issue and I will work on adding more details and approach once the main PR is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Handle segment flush threshold for commit time compaction enabled tables [pinot]
Posted by "tarun11Mavani (via GitHub)" <gi...@apache.org>.
tarun11Mavani closed issue #16632: Handle segment flush threshold for commit time compaction enabled tables
URL: https://github.com/apache/pinot/issues/16632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Handle segment flush threshold for commit time compaction enabled tables [pinot]
Posted by "tarun11Mavani (via GitHub)" <gi...@apache.org>.
tarun11Mavani commented on issue #16632:
URL: https://github.com/apache/pinot/issues/16632#issuecomment-3316706230
This is not needed anymore. Earlier I was testing my commit time compaction changes(#16344 and #16769) on 1.2 release branch. Since then, we have started using a new `SizeBasedSegmentFlushThresholdComputer` and removed usage of `SegmentFlushThresholdComputer` in #15347. Since I didn't have this change in 1.2 release branch in performance testing, I encountered the frequent commit issue. Any branch with 1.3+ release don't have this issue.
Below is example of three tables with commit time compaction disable, enabled and enabled+column major build (consuming from same kafka topic).
<img width="1315" height="115" alt="Image" src="https://github.com/user-attachments/assets/43d64b1f-4ea3-450a-a50a-fc2f5345cf10" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org