You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xingyingone (via GitHub)" <gi...@apache.org> on 2024/01/02 02:49:08 UTC
[I] [Bug] Memory leak of be [doris]
xingyingone opened a new issue, #29378:
URL: https://github.com/apache/doris/issues/29378
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
doris-2.0.4-rc02
### What's Wrong?
1. I execute some large query, be consumes a lot of memory (150G)
2. after I stop wirte or search, and the The doris cluster has no load, the memory does not decrease
### What You Expected?
when the doris cluster has no load, the memory should decrease
### How to Reproduce?
I don't known
### Anything Else?
no
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1908132311
> This is definitely a memory leak. After the above import, I tried creating smaller dump files and the BE servers immediately generated OOM becuase the prior leak had not been resolved. Only restarting the BE process seems to clear it. This appears to me to be a pretty substantial bug. No one else is experiencing this? Is there anything I can do to move this forward? Trying to build a POC but if I can get past this, we'll need to abandon and look at other options.
@seanfulton
Print heap profile when error `Allocator sys memory check failed` is reported,
```
1. Change prof:false of JEMALLOC_CONF in be.conf to prof:true and restart BE
2. curl http://be_host:be_webport/jeheap/dump
3. jeprof --dot lib/doris_be heap_dump_file_1
4. paste it to the http://www.webgraphviz.com/
```
Refer to https://doris.apache.org/community/developer-guide/debug-tool/#jemalloc-1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-3287227961
We're closing this issue because it hasn't been updated in a while.
If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "liujunfei2230 (via GitHub)" <gi...@apache.org>.
liujunfei2230 commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1884115253
i match the same problem on 2.0.3 version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1873798489
Hi @xingyingone, can the problem be reproduced? If possible, refer to https://doris.apache.org/zh-CN/community/developer-guide/debug-tool/#jemalloc-1 to print the jemalloc heap profile, and we will analyze it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1877034740
@sepastian
grep "Memory Tracker Summary" be.INFO, find the last log before BE OOM time, you will see the BE memory statistics at that time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "seanfulton (via GitHub)" <gi...@apache.org>.
seanfulton commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1881345553
Is there any update on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1877026070
> Unfortunately, it can't be reproduced at the moment
@xingyingone Try reducing segment cache capacity, add `segment_cache_capacity=10000` to `be.conf`.
The problem of memory not free in Doris 2.0.x is often caused by the segment cache being too large, because the segment cache is not accurately tracked in Doris 2.0.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "xingyingone (via GitHub)" <gi...@apache.org>.
xingyingone commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1873902586
Unfortunately, it can't be reproduced at the moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "seanfulton (via GitHub)" <gi...@apache.org>.
seanfulton commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1883564062
This is definitely a memory leak. After the above import, I tried creating smaller dump files and the BE servers immediately generated OOM becuase the prior leak had not been resolved. Only restarting the BE process seems to clear it.
This appears to me to be a pretty substantial bug.
No one else is experiencing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1908133236
> i match the same problem on 2.0.3 version
@liujunfei2230 Same as the previous comment: https://github.com/apache/doris/issues/29378#issuecomment-1908132311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "seanfulton (via GitHub)" <gi...@apache.org>.
seanfulton commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1875411260
I am seeing the same thing. We have a six-node cluster, three BE and three FE. All are 64G of RAM. We are loading a series of MySQL tables from files. The largest table is 1.5G. The files are a series of INSERT statements.
The BE nodes had 64G of RAM and we consistently ran into the limit of 52 G. I increased the RAM to 96G and now we are getting the error below. Restarting the BE process causes the RAM to drop, but once we restart the load, the RAM creeps up until it exceeds the limit.
ON the
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:27.237190 19832 daemon.cpp:262] [MemoryGC] start minor GC, process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory 18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:27.237272 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free process top memory overcommit query, , start seek all query, running query and load num: 63
I0103 08:02:27.237390 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free process top memory overcommit query, seek finished, seek 63 tasks. among them, 0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being canceled and has not been completed yet;
I0103 08:02:27.237396 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:27.237417 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free memory 0. cost(us): 192, details: :
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:27.793373 20963 olap_server.cpp:1065] cooldown producer get tablet num: 0
I0103 08:02:28.338559 19832 daemon.cpp:262] [MemoryGC] start minor GC, process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory 18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:28.338682 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free process top memory overcommit query, , start seek all query, running query and load num: 63
I0103 08:02:28.338913 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free process top memory overcommit query, seek finished, seek 63 tasks. among them, 0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being canceled and has not been completed yet;
I0103 08:02:28.338925 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:28.338955 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free memory 0. cost(us): 339, details: :
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:29.440176 19832 daemon.cpp:262] [MemoryGC] start minor GC, process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory 18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:29.440274 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free process top memory overcommit query, , start seek all query, running query and load num: 63
I0103 08:02:29.440418 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free process top memory overcommit query, seek finished, seek 63 tasks. among them, 0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being canceled and has not been completed yet;
I0103 08:02:29.440428 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:29.440454 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free memory 0. cost(us): 237, details: :
FreeTopOvercommitMemoryQuery:
WorkloadGroup:
I0103 08:02:30.061405 21622 heartbeat_server.cpp:61] get heartbeat from FE.host:10.10.21.11, port:9020, cluster id:1280404001, counter:9637, BE start time: 1704238503931
I0103 08:02:30.541870 19832 daemon.cpp:262] [MemoryGC] start minor GC, process memory used 72.75 GB exceed soft limit 69.12 GB or sys available memory 18.70 GB less than warning water mark 3.20 GB..
I0103 08:02:30.541988 19832 mem_tracker_limiter.cpp:522] [MemoryGC] GC free process top memory overcommit query, , start seek all query, running query and load num: 63
I0103 08:02:30.542172 19832 mem_tracker_limiter.cpp:560] [MemoryGC] GC free process top memory overcommit query, seek finished, seek 63 tasks. among them, 0 tasks can be canceled; 63 small tasks that were skipped; 0 tasks is being canceled and has not been completed yet;
I0103 08:02:30.542183 19832 mem_tracker_limiter.cpp:568] [MemoryGC] GC free process top memory overcommit query, finished, no task need be canceled.
I0103 08:02:30.542214 19832 mem_info.cpp:128] [MemoryGC] end minor GC, free memory 0. cost(us): 292, details: :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "seanfulton (via GitHub)" <gi...@apache.org>.
seanfulton commented on issue #29378:
URL: https://github.com/apache/doris/issues/29378#issuecomment-1877251253
I updated be.conf with the segment_cache_capacity=10000 and restarted the three BE nodes. Still getting the same error in the same place. Again, these are relatively small files with mysql insert statements generated by mysqldump.
I have a shell script that pipes the contents of the file into mysql -P9030, so each table is a new connection to doris. This is happening on every single file, all but one are less than 1 G
Error:
0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> >) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_6
Done with web_device.sql!
Loading web_os.sql ...
ERROR 1105 (HY000) at line 35: errCode = 2, detailMessage = (10.10.21.14)[MEM_ALLOC_FAILED]Create Expr failed because [E11] Allocator sys memory check failed: Cannot alloc:64, consuming tracker:<Load#Id=ba3be
2681a0d4635-8bea3f5473715482>, peak used 0, current used 0, exec node:<>, process memory used 77.24 GB exceed limit 76.80 GB or sys available memory 18.85 GB less than low water mark 1.60 GB.
0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> >) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_6
Done with web_os.sql!
Loading web_referrer.sql ...
ERROR 1105 (HY000) at line 35: errCode = 2, detailMessage = (10.10.21.14)[MEM_ALLOC_FAILED]Create Expr failed because [E11] Allocator sys memory check failed: Cannot alloc:64, consuming tracker:<Load#Id=e4aa2
acb46e94cd9-8906c1b443022160>, peak used 0, current used 0, exec node:<>, process memory used 77.24 GB exceed limit 76.80 GB or sys available memory 18.85 GB less than low water mark 1.60 GB.
0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> >) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_6
Done with web_referrer.sql!
Loading web_segment.sql ...
ERROR 1105 (HY000) at line 35: errCode = 2, detailMessage = (10.10.21.14)[MEM_ALLOC_FAILED]Create Expr failed because [E11] Allocator sys memory check failed: Cannot alloc:64, consuming tracker:<Load#Id=ae5c5
3468384e87-9bf5c7ef84169cf4>, peak used 0, current used 0, exec node:<>, process memory used 77.24 GB exceed limit 76.80 GB or sys available memory 18.85 GB less than low water mark 1.60 GB.
0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> >) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64
Done with web_segment.sql!
Loading web_social.sql ...
ERROR 1105 (HY000) at line 35: errCode = 2, detailMessage = (10.10.21.14)[MEM_ALLOC_FAILED]Create Expr failed because [E11] Allocator sys memory check failed: Cannot alloc:64, consuming tracker:<Load#Id=550b7
cabf404250-940d301bb7dca3be>, peak used 0, current used 0, exec node:<>, process memory used 77.24 GB exceed limit 76.80 GB or sys available memory 18.85 GB less than low water mark 1.60 GB.
0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> >) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Memory leak of be [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed issue #29378: [Bug] Memory leak of be
URL: https://github.com/apache/doris/issues/29378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org