You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/03/02 09:03:25 UTC

[GitHub] [incubator-doris] demon-gu opened a new issue #5442: Doris内存会不停增长

demon-gu opened a new issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442


   使用Doris时在be.conf中设置mem_limit，但是并不起作用，Doris使用的内存远远超过设置的百分比，直到被系统杀死be进程
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] killxdcj commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
killxdcj commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-793678221


   > 目前memtracker缺乏了一些内存的记录， 存在一些没有记录的内存空洞，后续我们会添加更详细的记录，调整一些不合理的内存配置来排查和规避这个问题。
   
   这块优化有详细点的计划或者有相关的讨论群组之类的嘛，我们也遇到了这个问题，目前也正在考虑投入精力优化下这块~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] demon-gu commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
demon-gu commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-793692299


   > > 我的Doris启动了许多Routine Load任务，是否是因为这些任务导致我be的内存一直增加，直到被系统kill呀？
   > 
   > 我是因为每5分钟执行一批Insert into select导致的
   
   我觉得每五分钟insert或者select是很正常的操作啊，我这边也经常有这种操作


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] HappenLee commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
HappenLee commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-789826855


   > 我也遇到了相同的问题，问题的场景是5分钟定时跑3-5个Insert into select 语句，每次执行完之后就看到内存往上跳跃性增长
   
   如果能稳定复现了话，麻烦把case给我提供一下。我的wechat:lchlhp472825648 @dongxl6 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] acelyc111 commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-790241903


   0.13可以在BE的web上看看内存消耗 ip:8040/mem_tracker


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] ZhaoBQ commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
ZhaoBQ commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-790241216


   我们也遇到相同问题，当从hdfs load数据的时候，内存使用会暴涨，然后be就挂了。有时候机器内存还有剩余，be没有挂就hang着。be相关日志：
   ![image](https://user-images.githubusercontent.com/35092554/109901393-f24cdf80-7cd3-11eb-98e3-173d0d4812dd.png)
   Doris版本是Apache Doris 0.13.0，be一共15台机器，每台内存188G。


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] HappenLee commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
HappenLee commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-793706489


   @killxdcj 最近会提一个pr梳理当前memtracker和调整的参数，你可以关注一下。有兴趣加我微信的私聊哈~~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] acelyc111 commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-790654610


   > > 0.13可以在BE的web上看看内存消耗 ip:8040/mem_tracker
   > 
   > ip:8040/memz吧
   
   mem_tracker是0.13新增的，能看到更详细内存（当然这里只捕获了其中部分内存使用情况）


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] EmmyMiao87 commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-796594110


   Could you please use both English and Chinese to explain this problem.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] ZhaoBQ commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
ZhaoBQ commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-790281240


   从metrics监控上看，内存使用不是很高，be hang住的时间点，内存使用并不高。我们的表一天八亿条数据，在hive里存着一天5G左右的大小，同时起两个任务分别导入一天的数据就会出问题，一天一天的导那就没有问题。
   ![image](https://user-images.githubusercontent.com/35092554/109911557-444a3100-7ce5-11eb-80f7-357d0dffdba2.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] morningman closed issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
morningman closed issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] demon-gu commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
demon-gu commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-788763850


   > which version are you using now.
   
   Server version: 5.1.0 Baidu Doris(Palo) version 0.13.15-Unknown


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] dongxl6 commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
dongxl6 commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-793689227


   > 我的Doris启动了许多Routine Load任务，是否是因为这些任务导致我be的内存一直增加，直到被系统kill呀？
   
   我是因为每5分钟执行一批Insert into select导致的


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] HappenLee commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
HappenLee commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-796436273


   各位，问题大概率定位到了是压缩算法部分存在比较恶性的bug，可能会导致数据导入和compaction时的内存占用可能会有数千倍的内存膨胀问题。
   
   ![image](https://user-images.githubusercontent.com/10553413/110734932-c1d0ec80-8263-11eb-849b-a08050e4f2c0.png)
   
   ![image](https://user-images.githubusercontent.com/10553413/110735000-e927b980-8263-11eb-8a5d-85fa35937f2b.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] ZhaoBQ commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
ZhaoBQ commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-791101786


   > > > 0.13可以在BE的web上看看内存消耗 ip:8040/mem_tracker
   > > 
   > > 
   > > ip:8040/memz吧
   > 
   > mem_tracker是0.13新增的，能看到更详细内存（当然这里只捕获了其中部分内存使用情况）
   
   mem_tracker好像没有，页面显示Not Found。
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] dongxl6 commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
dongxl6 commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-793683727


   12G机器，我把storage_page_cache_limit 修改为6-7G，目前看来是解决了。从源码看这个默认值好像是80G


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] dongxl6 commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
dongxl6 commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-790243017


   
   
   
   > 0.13可以在BE的web上看看内存消耗 ip:8040/mem_tracker
   
   ip:8040/memz吧


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] HappenLee commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
HappenLee commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-791114562


   目前memtracker缺乏了一些内存的记录， 存在一些没有记录的内存空洞，后续我们会添加更详细的记录，调整一些不合理的内存配置来排查和规避这个问题。


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] dongxl6 commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
dongxl6 commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-789617371


   我也遇到了相同的问题，问题的场景是5分钟定时跑3-5个Insert into select 语句，每次执行完之后就看到内存往上跳跃性增长


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] HappenLee commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
HappenLee commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-788742381


   which version are you using now.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] demon-gu commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
demon-gu commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-793687886


   我的Doris启动了许多Routine Load任务，是否是因为这些任务导致我be的内存一直增加，直到被系统kill呀？


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] ZhaoBQ commented on issue #5442: Doris内存会不停增长

Posted by GitBox <gi...@apache.org>.
ZhaoBQ commented on issue #5442:
URL: https://github.com/apache/incubator-doris/issues/5442#issuecomment-791111069


   原来通过broker load两天数据，be都会挂，连续复现了两次。注释掉mmap相关配置重启be后导入3天目前没什么问题。配置如下：
   `
   #mmap_buffers=true
   #use_mmap_allocate_chunk=true
   #enable_quadratic_probing=true
   `
   目前导入3天数据，部分机器内存使用量也挺高的，Doris是把所以数据先放在内存里，然后统一刷下去？
   ![image](https://user-images.githubusercontent.com/35092554/110059898-1970e300-7da0-11eb-8ea9-cd24853093f6.png)
   ![image](https://user-images.githubusercontent.com/35092554/110060014-360d1b00-7da0-11eb-816a-85fa3a02f52c.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org