[GitHub] jihoonson opened a new issue #6286: Too many open files error in indexTask

[GitBox <gi...@apache.org>]

jihoonson opened a new issue #6286: Too many open files error in indexTask
URL: https://github.com/apache/incubator-druid/issues/6286
 
 
   `AppenderatorImpl` holds all generated segment files until `abandonSegments()` is called. `IndexTask` works based on `AppenderatorImpl` and calls `abandonSegments()` once it pushes the segments to deep storage. This can cause an issue of too many open files if the indexTask is running with the perfect rollup mode. In the perfect rollup mode, the indexTask can't push segments in the middle of indexing. Instead, they should hold all segments until the very end of the indexing. 
   
   The way how `AppenderatorImpl` works makes sense for stream ingestion because the ingested data should be available for query immediately. However, batch tasks don't have to hold them.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org