You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/05/11 13:50:48 UTC

[GitHub] [incubator-doris] JinLiOnline opened a new issue #5794: [BE OOM] be使用的内存远超操作系统内存,pagetable过大,引发oom

JinLiOnline opened a new issue #5794:
URL: https://github.com/apache/incubator-doris/issues/5794


   **Describe the bug**
   doris13版本,BE 发生oom, 操作系统可用内存700G,top命令中显示,be只占用600G,其他进程占用内存很小。但是be却发生oom,被系统kill掉。
   
   经过排查发现,查看be的8040端口  
   $ curl http://127.0.0.1:8040/memz
   
   <pre>Mem Limit: 603.26 GB
   Mem Consumption: 29.70 GB
   
   MALLOC:   40138085404224 (38278661.2 MiB) Bytes in use by application
   MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
   MALLOC: +  19226145608 (18335.5 MiB) Bytes in central cache freelist
   MALLOC: +       262144 (    0.2 MiB) Bytes in transfer cache freelist
   MALLOC: +    843947640 (  804.9 MiB) Bytes in thread cache freelists
   MALLOC: +  42186440704 (40232.1 MiB) Bytes in malloc metadata
   MALLOC:   ------------
   MALLOC: = 40200342200320 (38338033.9 MiB) Actual memory used (physical + swap)
   MALLOC: + 1512264359936 (1442207.7 MiB) Bytes released to OS (aka unmapped)
   MALLOC:   ------------
   MALLOC: = 41712606560256 (39780241.5 MiB) Virtual address space used
   MALLOC:
   MALLOC:       28652164              Spans in use
   MALLOC:           1010              Thread heaps in use
   MALLOC:           8192              Tcmalloc page size
   
   Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
   Bytes released to the OS take up virtual address space but no physical memory.
   
   
   上面显示malloc 38278661.2MB ==> 36T, 远超物理内存。
   
   查看pagetables发现:
   
   $ cat /proc/meminfo | grep PageTables
   PageTables:     79106020 kB
   
   换算得 75G。
   
   pagetables吃掉 75G,be占用600G,导致操作系统内存不足,触发oom干掉消耗内存最大的 be进程。
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] JinLiOnline commented on issue #5794: [BE OOM] be使用的内存远超操作系统内存,pagetable过大,引发oom

Posted by GitBox <gi...@apache.org>.
JinLiOnline commented on issue #5794:
URL: https://github.com/apache/incubator-doris/issues/5794#issuecomment-839374031


   > BTW, have you enabled swap?
   disable swap
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] xinghuayu007 commented on issue #5794: [BE OOM] be使用的内存远超操作系统内存,pagetable过大,引发oom

Posted by GitBox <gi...@apache.org>.
xinghuayu007 commented on issue #5794:
URL: https://github.com/apache/incubator-doris/issues/5794#issuecomment-839353307


   top -Hp pid观察下发生OOM时那些线程在工作,我们最近升级到0.13版本,发生了频繁OOM,大部分原因是compaction导致的,通过调参抑制base compaction and cumulative compaction得到缓解。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] JinLiOnline commented on issue #5794: [BE OOM] be使用的内存远超操作系统内存,pagetable过大,引发oom

Posted by GitBox <gi...@apache.org>.
JinLiOnline commented on issue #5794:
URL: https://github.com/apache/incubator-doris/issues/5794#issuecomment-838557207


   请问,tcmalloc占用这么高,一般是什么问题导致的,线上集群不太好profile,如何进一步定位?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman commented on issue #5794: [BE OOM] be使用的内存远超操作系统内存,pagetable过大,引发oom

Posted by GitBox <gi...@apache.org>.
morningman commented on issue #5794:
URL: https://github.com/apache/incubator-doris/issues/5794#issuecomment-838735453


   38T is the memory requested by tcmalloc but not the memory actually used.
   BTW, have you enabled swap?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org