You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@doris.apache.org by "weilai201 (via GitHub)" <gi...@apache.org> on 2024/03/01 03:29:32 UTC

[I] [Bug] doris通过datax导入数据提示内存超限 [doris]

weilai201 opened a new issue, #31637:
URL: https://github.com/apache/doris/issues/31637

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Version
   
   2.0.4-rc6
   
   ### What's Wrong?
   
   我现在有一台4和16GB内存的机器，上面部署了doris fe和be。be做了如下配置：
   mem_limit = 50%
   segment_cache_capacity=10000
   disable_mem_pools=true
   load_process_max_memory_limit_percent=40
   
   在使用datax持续写入数据过程中，发现be内存一直在增加，直到系统datax导入数据失败。
   错误消息如下
   `{"BeginTxnTimeMs":0,"Comment":"","CommitAndPublishTimeMs":0,"Label":"my_prefix291e78fc-ad72-4fdb-9d59-c28a8d04519a_45","LoadBytes":8376020,"LoadTimeMs":3146,"Message":"[INTERNAL_ERROR]cancelled: Process has no memory available, cancel top memory used load: load memory tracker <Load#Id=9f448e1aefe9d246-9a7f4d671b5aeeb6> consumption 11.09 MB, backend 192.168.1.103 process memory used 7.74 GB exceed limit 7.69 GB or sys available memory 1.46 GB less than low water mark 1.54 GB. Execute again after enough memory, details see be.INFO.\n\n\t0#  doris::io::StreamLoadPipe::_append(std::shared_ptr<doris::ByteBuffer> const&, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0\n\t1#  doris::io::StreamLoadPipe::append(std::shared_ptr<doris::ByteBuffer> const&) at /home/zcp/repo_center/doris_release/doris/be/src/io/fs/stream_load_pipe.cpp:162\n\t2#  doris::StreamLoadAction::on_chunk_data(doris::HttpRequest*) at /home/zcp/repo_center/doris_release/doris/be/src/
 common/status.h:442\n\t3#  ?\n\t4#  bufferevent_run_readcb_\n\t5#  ?\n\t6#  ?\n\t7#  ?\n\t8#  ?\n\t9#  std::_Function_handler<void (), doris::EvHttpServer::start()::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98\n\t10# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0\n\t11# doris::Thread::supervise_thread(void*) at /var/local/ldb_toolchain/bin/../usr/include/pthread.h:562\n\t12# start_thread\n\t13# clone\n","NumberFilteredRows":0,"NumberLoadedRows":0,"NumberTotalRows":0,"NumberUnselectedRows":0,"ReadDataTimeMs":17,"Status":"Fail","StreamLoadPutTimeMs":3,"TwoPhaseCommit":"false","TxnId":2124,"WriteDataTimeMs":1918}
    `
   
   内存消耗截图：
   
   <img width="1837" alt="image" src="https://github.com/apache/doris/assets/12964080/e0b9d2a5-4e31-463a-bbb3-52f446cad88c">
   
   
   ### What You Expected?
   
   be可以自动做内存释放！当内存不足时，可降低处理速度，提升系统稳定性
   
   ### How to Reproduce?
   
   _No response_
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] doris通过datax导入数据提示内存超限 [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.

morningman closed issue #31637: [Bug] doris通过datax导入数据提示内存超限
URL: https://github.com/apache/doris/issues/31637


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] doris通过datax导入数据提示内存超限 [doris]

Posted by "zhbdesign (via GitHub)" <gi...@apache.org>.

zhbdesign commented on issue #31637:
URL: https://github.com/apache/doris/issues/31637#issuecomment-1972655156

   workload group现在可以限制查询，可以排队，但是streamload不知道能不能排队


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] doris通过datax导入数据提示内存超限 [doris]

Posted by "zhbdesign (via GitHub)" <gi...@apache.org>.

zhbdesign commented on issue #31637:
URL: https://github.com/apache/doris/issues/31637#issuecomment-1972781704

   但是maxBatchRows也是满足条件就插入，这个可以搞得大一点，doris不怕一个批次大，就怕插入快


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Bug] doris通过datax导入数据提示内存超限 [doris]

Posted by "zhbdesign (via GitHub)" <gi...@apache.org>.

zhbdesign commented on issue #31637:
URL: https://github.com/apache/doris/issues/31637#issuecomment-1973114231

   我也是遇到相同的问题，调整导入频率就可以解决，doris本身应该没什么再调节的了，你可以试试单副本导入
   https://mp.weixin.qq.com/s/aLq7glxsN4NWw01RS_7riA


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org