[GitHub] [doris] yiguolei commented on a diff in pull request #10285: force to flush memtable when loadchannel mem exceeds

[GitBox <gi...@apache.org>]

yiguolei commented on code in PR #10285:
URL: https://github.com/apache/doris/pull/10285#discussion_r903185292


##########
be/src/runtime/load_channel_mgr.cpp:
##########
@@ -177,34 +179,46 @@ Status LoadChannelMgr::add_batch(const PTabletWriterAddBatchRequest& request,
 void LoadChannelMgr::_handle_mem_exceed_limit() {
     // lock so that only one thread can check mem limit
     std::lock_guard<std::mutex> l(_lock);
-    if (!_mem_tracker->limit_exceeded()) {
-        return;
-    }
-
-    int64_t max_consume = 0;
-    std::shared_ptr<LoadChannel> channel;
-    for (auto& kv : _load_channels) {
-        if (kv.second->is_high_priority()) {
-            // do not select high priority channel to reduce memory
-            // to avoid blocking them.
+    while (_mem_tracker->limit_exceeded()) {
+        LOG(INFO) << "LoadChannelMgr consume " << _mem_tracker->consumption()/(1024*1024) <<"M bytes, exceeded limit" << _mem_tracker->limit();
+        if (StorageEngine::instance()->memtable_flush_executor()->thread_pool_overloaded()) {

Review Comment:
   这个逻辑会让当前正在写入的线程卡死。 因为另外一个写入线程可能 _mem_tracker->limit_exceeded() 为false，但是不停的往flush queue里加memtable，那么当前这个就会一直卡死了。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org