[GitHub] [incubator-pinot] siddharthteotia commented on issue #5610: Direct Memory OOM issue during upgrade to 0.4.0

[GitBox <gi...@apache.org>]

siddharthteotia commented on issue #5610:
URL: https://github.com/apache/incubator-pinot/issues/5610#issuecomment-648514422


   @fx19880617 , do we have the segments handy? We can load the segment and look into the file header to see number of documents per chunk. I had made an improvement (as part of supporting text blobs for text index) by deriving the number of documents per chunk based on the length of data as opposed to hard coded 1000. So if we pack more than 1000 rows per chunk, the size of each chunk will increase. 
   
   I don't think that would be the issue since I had this PR https://github.com/apache/incubator-pinot/pull/5470 which was included in the 0.4.0 release (a new RC was cut by Haibo to include this) and actually made this behavior configurable (disabled by default) 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org