You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/09/18 14:36:15 UTC

[GitHub] [incubator-doris] morningman opened a new issue #6701: [Bug] BE out of memory after loading

morningman opened a new issue #6701:
URL: https://github.com/apache/incubator-doris/issues/6701


   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Version
   
   trunk fee8e6a
   
   ### What's Wrong?
   
   BE OOM and crash after loading a large volume of data.
   For example, there is only 1 tablet in a table, and if we loading 100GB data in one broker load job.
   After loading finished, the OOM will happen.
   
   This is because there are too many segments in one rowset, so that when doing compaction for this rowset,
   it will take too much memory, which cause OOM.
   
   ### What You Expected?
   
   Don't crash.
   
   ### How to Reproduce?
   
   Loading a large volume of data in one load job to a table with a few tablets.
   
   ### Anything Else?
   
   I am about to limit the number of segments in the newly created segment.
   If the number of segment exceed limit, load will failed with error code -238.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] morningman closed issue #6701: [Bug] BE out of memory after loading

Posted by GitBox <gi...@apache.org>.
morningman closed issue #6701:
URL: https://github.com/apache/incubator-doris/issues/6701


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org