[GitHub] [incubator-doris] kangpinghuang opened a new issue #2712: [SegmentV2] optimize fd management in Segment V2

[GitBox <gi...@apache.org>]

kangpinghuang opened a new issue #2712: [SegmentV2] optimize fd management in Segment V2
URL: https://github.com/apache/incubator-doris/issues/2712
 
 
   In a situation with many load jobs, there are risks that BE will core dump because of Too Many Open File error.
   The reason is that in segment v2, the Segment will hold fd before the rowset is closed. When there are many load jobs, the compacted rowset can not be deleted for incremental rowsets will hold them for 30min, although compactions are quick enough.
   So we should add a mechanism to close the fd when we finished read file and add a fd cache for performance.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org