[GitHub] [incubator-pinot] dongxiaoman opened a new issue #6275: Performance improvement needed when pushing segments for table with lots of segments

[GitBox <gi...@apache.org>]

dongxiaoman opened a new issue #6275:
URL: https://github.com/apache/incubator-pinot/issues/6275


   Right now if we have 100k+ segments, pushing new segments into the table is very slow. 
   
   **In general, we need to find a strategy to make pushing lots of segments faster**
   
   Below are some ideas for tracking:
   
   One possible direct reason is that the update on IdealState of segments will be very slow.
   
   By checking jstack snapshots, I can see that when we update IdealState, we try to clone the IdealState and then compare the new value with old value.
   
   This is fine for small IdealStates, but currently the segment assignment states are being cloned by "Serializing => (Json) => Deserializing" which maybe is the reason why it is slow.
   
   Another idea is maybe we could have a segment push queue, which merges the segment push results and do a batch update on ideal states.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org