You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "TimChen44 (via GitHub)" <gi...@apache.org> on 2023/04/14 08:45:07 UTC
[GitHub] [doris] TimChen44 opened a new issue, #18677: [Bug] 当处理一个非常大的数据时，发生内存溢出，并且be掉线
TimChen44 opened a new issue, #18677:
URL: https://github.com/apache/doris/issues/18677
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
1.2.3
### What's Wrong?
当我处理十几亿数据的时候出现了内存移除，并且be节点掉线并停止服务，下面时错误信息
I0414 16:26:50.484669 21338 daemon.cpp:289] Process regular GC Cache, Free Memory 0 Bytes
W0414 16:26:55.842762 22006 beta_rowset_writer.cpp:950] too many segments in rowset. tablet_id:45362 rowset_id:0200000000000041554b66730bc2563eccaa43479f792cb5 max:200 _num_segment:200 _segcompacted_point:0 _num_segcompacted:0
W0414 16:26:55.901852 22006 merger.cpp:170] failed to write block when merging rowsets of tablet 45362.954735494.85477ec08198b797-b0869f63c6cff1b4[res=[E-238]]
W0414 16:26:55.902809 22006 compaction.cpp:351] fail to do vcumulative compaction. res=[E-238], tablet=45362.954735494.85477ec08198b797-b0869f63c6cff1b4, output_version=[296-381]
W0414 16:26:55.902848 22006 tablet.cpp:1596] failed to do cumulative compaction. res=[E-238], tablet=45362.954735494.85477ec08198b797-b0869f63c6cff1b4
I0414 16:27:02.334146 22007 compaction.cpp:320] start vcumulative compaction. tablet=45362.954735494.85477ec08198b797-b0869f63c6cff1b4, output_version=[296-381], permits: 100
I0414 16:27:03.209165 22008 compaction.cpp:400] succeed to do vcumulative compaction is_vertical=0. tablet=49060.1136312067.ab42e1a65125268f-63fc503d24052c8a, output_version=[138-167], current_max_version=199, disk=/opt/doris/apache-doris-be-1.2.3-bin-x86_64/storage, segments=103, input_row_num=150000000, output_row_num=150000000. elapsed time=990.879s. cumulative_compaction_policy=SIZE_BASED, compact_row_per_second=151380
I0414 16:27:03.306368 22008 compaction.cpp:320] start vcumulative compaction. tablet=49060.1136312067.ab42e1a65125268f-63fc503d24052c8a, output_version=[184-199], permits: 5
### What You Expected?
当发生内存溢出时，当前查询请求可以失败，但是be节点不能掉线，服务不能停止，在释放本次查询使用的内存后，可以继续为其他查询提供服务
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] JayITH commented on issue #18677: [Bug] 当处理一个非常大的数据时，发生内存溢出，并且be掉线
Posted by "JayITH (via GitHub)" <gi...@apache.org>.
JayITH commented on issue #18677:
URL: https://github.com/apache/doris/issues/18677#issuecomment-1512553632
you need set max_segment_num_per_rowset > 200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TimChen44 closed issue #18677: [Bug] 当处理一个非常大的数据时，发生内存溢出，并且be掉线
Posted by "TimChen44 (via GitHub)" <gi...@apache.org>.
TimChen44 closed issue #18677: [Bug] 当处理一个非常大的数据时，发生内存溢出，并且be掉线
URL: https://github.com/apache/doris/issues/18677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org