You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2026/01/29 06:55:29 UTC

[PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

bobhan1 opened a new pull request, #60344:
URL: https://github.com/apache/doris/pull/60344

   ### What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3815832759

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4388226608

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153568462

   # FE UT Coverage Report
   Increment line coverage `  100.00% (10/10)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_e11f6697a68de51291f93879176dc7822361e9ed/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_e11f6697a68de51291f93879176dc7822361e9ed/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60344:
URL: https://github.com/apache/doris/pull/60344#discussion_r2925187722


##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -423,6 +504,10 @@ Status retry_rpc(std::string_view op_name, const Request& req, Response* res,
         } else if (res->status().code() == MetaServiceCode::INVALID_ARGUMENT) {
             return Status::Error<ErrorCode::INVALID_ARGUMENT, false>("failed to {}: {}", op_name,
                                                                      res->status().msg());
+        } else if (res->status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
+            // TODO(bobhan1): change to correct error code
+            rate_limit_ctx.backpressure_handler->on_ms_busy();

Review Comment:
   **Bug (Critical): Null pointer dereference**
   
   `rate_limit_ctx.backpressure_handler` can be `nullptr`. The `RpcRateLimitCtx` struct defaults `backpressure_handler` to `nullptr`, and `get_cluster_status` (line ~2516) passes `{.host_limiters = host_level_ms_rpc_rate_limiters_}` without setting `.backpressure_handler`. If MS returns `MAX_QPS_LIMIT` for that RPC, this will crash.
   
   Both `apply_rate_limit()` (line 418) and `record_rpc_qps()` (line 446) correctly null-check `ctx.backpressure_handler` before use. The same guard is needed here.
   
   Suggested fix:
   ```cpp
   } else if (res->status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
       if (rate_limit_ctx.backpressure_handler) {
           rate_limit_ctx.backpressure_handler->on_ms_busy();
       }
       // MS_BUSY should also be retried
   ```



##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -631,6 +727,12 @@ Status CloudMetaMgr::sync_tablet_rowsets_unlocked(CloudTablet* tablet,
             return Status::NotFound("failed to get rowset meta: {}, {}", resp.status().msg(),
                                     tablet_info);
         }
+        if (resp.status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
+            // TODO(bobhan1): change to correct error code
+            ms_backpressure_handler_->on_ms_busy();

Review Comment:
   **Bug (Critical): Null pointer dereference + Infinite loop**
   
   Two issues here:
   
   **1. Null deref:** `ms_backpressure_handler_` is initialized to `nullptr` (see `cloud_meta_mgr.h`) and only set via `set_ms_backpressure_handler()`. If this setter was never called, or called with null, a `MAX_QPS_LIMIT` response will crash. Line 698 correctly null-checks `host_level_ms_rpc_rate_limiters_` in the same function — the backpressure handler should get the same treatment.
   
   **2. Infinite busy-loop:** The `continue` on line 734 skips incrementing `tried` and has no backoff/sleep. If MS keeps returning `MAX_QPS_LIMIT`, this is an unbounded busy-loop consuming 100% CPU. Contrast with `retry_rpc()` which increments `retry_times` and applies exponential backoff.
   
   Suggested fix:
   ```cpp
   if (resp.status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
       if (ms_backpressure_handler_) {
           ms_backpressure_handler_->on_ms_busy();
       }
       ++tried; // Count against retry limit
       // Apply backoff similar to retry_rpc
       bthread_usleep(std::min(1000 * (1 << tried), 60000) * 1000ULL);
       continue;
   }
   ```



##########
be/src/cloud/cloud_ms_backpressure_handler.cpp:
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_ms_backpressure_handler.h"
+
+#include <fmt/format.h>
+#include <glog/logging.h>
+
+#include <algorithm>
+#include <cmath>
+#include <queue>
+#include <thread>
+
+#include "cloud/config.h"
+#include "common/status.h"
+#include "util/thread.h"
+
+namespace doris::cloud {
+
+// Global bvar metrics
+bvar::Adder<uint64_t> g_backpressure_upgrade_count("ms_rpc_backpressure_upgrade_count");
+bvar::Window<bvar::Adder<uint64_t>> g_backpressure_upgrade_60s("ms_rpc_backpressure_upgrade_60s",
+                                                               &g_backpressure_upgrade_count, 60);
+bvar::Adder<uint64_t> g_backpressure_downgrade_count("ms_rpc_backpressure_downgrade_count");
+bvar::Window<bvar::Adder<uint64_t>> g_backpressure_downgrade_60s(
+        "ms_rpc_backpressure_downgrade_60s", &g_backpressure_downgrade_count, 60);
+bvar::LatencyRecorder g_throttle_wait_prepare_rowset(
+        "ms_rpc_backpressure_throttle_wait_prepare_rowset");
+bvar::LatencyRecorder g_throttle_wait_commit_rowset(
+        "ms_rpc_backpressure_throttle_wait_commit_rowset");
+bvar::LatencyRecorder g_throttle_wait_update_tmp_rowset(
+        "ms_rpc_backpressure_throttle_wait_update_tmp_rowset");
+bvar::LatencyRecorder g_throttle_wait_update_packed_file_info(
+        "ms_rpc_backpressure_throttle_wait_update_packed_file_info");
+bvar::LatencyRecorder g_throttle_wait_update_delete_bitmap(
+        "ms_rpc_backpressure_throttle_wait_update_delete_bitmap");
+bvar::Adder<uint64_t> g_ms_busy_count("ms_rpc_backpressure_ms_busy_count");
+bvar::Window<bvar::Adder<uint64_t>> g_ms_busy_60s("ms_rpc_backpressure_ms_busy_60s",
+                                                  &g_ms_busy_count, 60);
+
+static bvar::LatencyRecorder* s_throttle_wait_recorders[] = {
+        &g_throttle_wait_prepare_rowset,       &g_throttle_wait_commit_rowset,
+        &g_throttle_wait_update_tmp_rowset,    &g_throttle_wait_update_packed_file_info,
+        &g_throttle_wait_update_delete_bitmap,
+};
+
+bvar::LatencyRecorder* get_throttle_wait_recorder(LoadRelatedRpc rpc) {
+    size_t idx = static_cast<size_t>(rpc);
+    if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+        return nullptr;
+    }
+    return s_throttle_wait_recorders[idx];
+}
+
+// ============== StrictQpsLimiter ==============
+
+StrictQpsLimiter::StrictQpsLimiter(double qps) {
+    if (qps <= 0) {
+        qps = 1.0;
+    }
+    _interval_ns = static_cast<int64_t>(1e9 / qps);
+    _next_allowed_time = Clock::now();
+}
+
+StrictQpsLimiter::Clock::time_point StrictQpsLimiter::reserve() {
+    std::lock_guard lock(_mtx);
+    auto now = Clock::now();
+    if (_next_allowed_time <= now) {
+        _next_allowed_time = now + std::chrono::nanoseconds(_interval_ns);
+        return now;
+    }
+    auto result = _next_allowed_time;
+    _next_allowed_time += std::chrono::nanoseconds(_interval_ns);
+    return result;
+}
+
+void StrictQpsLimiter::update_qps(double new_qps) {
+    if (new_qps <= 0) {
+        new_qps = 1.0;
+    }
+    std::lock_guard lock(_mtx);
+    _interval_ns = static_cast<int64_t>(1e9 / new_qps);
+}
+
+double StrictQpsLimiter::get_qps() const {
+    std::lock_guard lock(_mtx);
+    if (_interval_ns <= 0) {
+        return 0;
+    }
+    return 1e9 / _interval_ns;
+}
+
+// ============== TableRpcQpsCounter ==============
+
+TableRpcQpsCounter::TableRpcQpsCounter(int64_t table_id, LoadRelatedRpc rpc_type, int window_sec)
+        : _table_id(table_id), _rpc_type(rpc_type) {
+    _counter = std::make_unique<bvar::Adder<int64_t>>();
+    _counter->hide();
+    _qps = std::make_unique<bvar::PerSecond<bvar::Adder<int64_t>>>(_counter.get(), window_sec);
+    _qps->hide();
+}
+
+void TableRpcQpsCounter::increment() {
+    (*_counter) << 1;
+}
+
+double TableRpcQpsCounter::get_qps() const {
+    return _qps->get_value();
+}
+
+// ============== TableRpcQpsRegistry ==============
+
+TableRpcQpsRegistry::TableRpcQpsRegistry() = default;
+
+void TableRpcQpsRegistry::record(LoadRelatedRpc rpc_type, int64_t table_id) {
+    auto* counter = get_or_create_counter(rpc_type, table_id);
+    if (counter) {
+        counter->increment();
+    }
+}
+
+TableRpcQpsCounter* TableRpcQpsRegistry::get_or_create_counter(LoadRelatedRpc rpc_type,
+                                                               int64_t table_id) {
+    size_t idx = static_cast<size_t>(rpc_type);
+    if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+        return nullptr;
+    }
+
+    {
+        std::shared_lock lock(_mutex);
+        auto it = _counters[idx].find(table_id);
+        if (it != _counters[idx].end()) {
+            return it->second.get();
+        }
+    }
+
+    std::unique_lock lock(_mutex);
+    // Double check after acquiring exclusive lock
+    auto it = _counters[idx].find(table_id);
+    if (it != _counters[idx].end()) {
+        return it->second.get();
+    }
+
+    auto counter = std::make_unique<TableRpcQpsCounter>(table_id, rpc_type,
+                                                        config::ms_rpc_table_qps_window_sec);
+    auto* ptr = counter.get();
+    _counters[idx][table_id] = std::move(counter);
+    return ptr;
+}
+
+std::vector<std::pair<int64_t, double>> TableRpcQpsRegistry::get_top_k_tables(
+        LoadRelatedRpc rpc_type, int k) const {
+    size_t idx = static_cast<size_t>(rpc_type);
+    if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT) || k <= 0) {
+        return {};
+    }
+
+    // Use a min-heap of size k to find top-k without allocating a vector for all tables.
+    // The heap top is the smallest among the k largest elements seen so far.
+    using Entry = std::pair<int64_t, double>; // (table_id, qps)
+    auto min_cmp = [](const Entry& a, const Entry& b) { return a.second > b.second; };
+    std::priority_queue<Entry, std::vector<Entry>, decltype(min_cmp)> min_heap(min_cmp);
+
+    {
+        std::shared_lock lock(_mutex);
+        for (const auto& [table_id, counter] : _counters[idx]) {
+            double qps = counter->get_qps();
+            if (qps > 0) {
+                if (static_cast<int>(min_heap.size()) < k) {
+                    min_heap.push({table_id, qps});
+                } else if (qps > min_heap.top().second) {
+                    min_heap.pop();
+                    min_heap.push({table_id, qps});
+                }
+            }
+        }
+    }
+
+    // Extract results from heap (comes out in ascending order, reverse to descending)
+    std::vector<Entry> result;
+    result.reserve(min_heap.size());
+    while (!min_heap.empty()) {
+        result.push_back(min_heap.top());
+        min_heap.pop();
+    }
+    std::reverse(result.begin(), result.end());
+
+    return result;
+}
+
+double TableRpcQpsRegistry::get_qps(LoadRelatedRpc rpc_type, int64_t table_id) const {
+    size_t idx = static_cast<size_t>(rpc_type);
+    if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+        return 0;
+    }
+
+    std::shared_lock lock(_mutex);
+    auto it = _counters[idx].find(table_id);
+    if (it != _counters[idx].end()) {
+        return it->second->get_qps();
+    }
+    return 0;
+}
+
+void TableRpcQpsRegistry::cleanup_inactive_tables() {
+    std::unique_lock lock(_mutex);
+
+    for (size_t idx = 0; idx < static_cast<size_t>(LoadRelatedRpc::COUNT); ++idx) {
+        auto& counter_map = _counters[idx];
+        for (auto it = counter_map.begin(); it != counter_map.end();) {
+            // Remove counters with zero QPS for a long time
+            if (it->second->get_qps() < 0.01) {
+                it = counter_map.erase(it);
+            } else {
+                ++it;
+            }
+        }
+    }
+}
+
+// ============== TableRpcThrottler ==============
+
+TableRpcThrottler::TableRpcThrottler() {
+    // Initialize bvar for throttled table counts
+    for (size_t i = 0; i < static_cast<size_t>(LoadRelatedRpc::COUNT); ++i) {
+        std::string bvar_name = fmt::format("ms_rpc_backpressure_throttled_tables_{}",
+                                            load_related_rpc_name(static_cast<LoadRelatedRpc>(i)));
+        _throttled_table_counts[i] = std::make_unique<bvar::Status<size_t>>(bvar_name, 0);
+    }
+}
+
+std::chrono::steady_clock::time_point TableRpcThrottler::throttle(LoadRelatedRpc rpc_type,
+                                                                  int64_t table_id) {
+    std::shared_lock lock(_mutex);
+    auto it = _limiters.find({rpc_type, table_id});
+    if (it == _limiters.end()) {
+        return std::chrono::steady_clock::now();
+    }
+    return it->second->reserve();
+}
+
+void TableRpcThrottler::set_qps_limit(LoadRelatedRpc rpc_type, int64_t table_id, double qps_limit) {
+    if (qps_limit <= 0) {
+        return;
+    }
+
+    std::unique_lock lock(_mutex);
+    auto key = std::make_pair(rpc_type, table_id);
+    auto it = _limiters.find(key);
+    if (it != _limiters.end()) {
+        it->second->update_qps(qps_limit);
+    } else {
+        _limiters[key] = std::make_unique<StrictQpsLimiter>(qps_limit);
+        // Update bvar count
+        size_t idx = static_cast<size_t>(rpc_type);
+        if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+            size_t count = 0;
+            for (const auto& [k, _] : _limiters) {
+                if (k.first == rpc_type) {
+                    ++count;
+                }
+            }
+            _throttled_table_counts[idx]->set_value(count);
+        }
+    }
+
+    LOG(INFO) << "[ms-throttle] set table QPS limit: rpc=" << load_related_rpc_name(rpc_type)
+              << ", table_id=" << table_id << ", qps_limit=" << qps_limit;
+}
+
+void TableRpcThrottler::remove_qps_limit(LoadRelatedRpc rpc_type, int64_t table_id) {
+    std::unique_lock lock(_mutex);
+    auto key = std::make_pair(rpc_type, table_id);
+    auto it = _limiters.find(key);
+    if (it != _limiters.end()) {
+        _limiters.erase(it);
+        // Update bvar count
+        size_t idx = static_cast<size_t>(rpc_type);
+        if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+            size_t count = 0;
+            for (const auto& [k, _] : _limiters) {
+                if (k.first == rpc_type) {
+                    ++count;
+                }
+            }
+            _throttled_table_counts[idx]->set_value(count);
+        }
+
+        LOG(INFO) << "[ms-throttle] removed table QPS limit: rpc="
+                  << load_related_rpc_name(rpc_type) << ", table_id=" << table_id;
+    }
+}
+
+double TableRpcThrottler::get_qps_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+    std::shared_lock lock(_mutex);
+    auto it = _limiters.find({rpc_type, table_id});
+    if (it != _limiters.end()) {
+        return it->second->get_qps();
+    }
+    return 0;
+}
+
+bool TableRpcThrottler::has_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+    std::shared_lock lock(_mutex);
+    return _limiters.find({rpc_type, table_id}) != _limiters.end();
+}
+
+size_t TableRpcThrottler::get_throttled_table_count(LoadRelatedRpc rpc_type) const {
+    size_t idx = static_cast<size_t>(rpc_type);
+    if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+        return 0;
+    }
+    return _throttled_table_counts[idx]->get_value();
+}
+
+std::vector<TableRpcThrottler::ThrottleEntry> TableRpcThrottler::get_all_throttled_entries() const {
+    std::shared_lock lock(_mutex);
+    std::vector<ThrottleEntry> entries;
+    entries.reserve(_limiters.size());
+    for (const auto& [key, limiter] : _limiters) {
+        entries.push_back({key.first, key.second, limiter->get_qps()});
+    }
+    return entries;
+}
+
+// ============== MSBackpressureHandler ==============
+
+MSBackpressureHandler::MSBackpressureHandler(TableRpcQpsRegistry* qps_registry,
+                                             TableRpcThrottler* throttler)
+        : _qps_registry(qps_registry),
+          _throttler(throttler),
+          _stop_latch(1),
+          _last_ms_busy_time(std::chrono::steady_clock::time_point::min()) {
+    // Initialize state machine with config values
+    RpcThrottleParams throttle_params {
+            .top_k = config::ms_backpressure_upgrade_top_k,
+            .ratio = config::ms_backpressure_throttle_ratio,
+            .floor_qps = config::ms_rpc_table_qps_limit_floor,
+    };
+    _state_machine = std::make_unique<RpcThrottleStateMachine>(throttle_params);
+
+    // Initialize coordinator with config values
+    // Coordinator uses ticks where 1 tick = 1 millisecond (fixed unit)
+    // This allows tick_interval_ms to change at runtime without affecting correctness
+    ThrottleCoordinatorParams coordinator_params {
+            .upgrade_cooldown_ticks = config::ms_backpressure_upgrade_interval_ms,
+            .downgrade_after_ticks = config::ms_backpressure_downgrade_interval_ms,
+    };
+    _coordinator = std::make_unique<RpcThrottleCoordinator>(coordinator_params);
+
+    auto st = Thread::create(
+            "MSBackpressureHandler", "tick_thread", [this]() { this->_tick_thread_callback(); },
+            &_tick_thread);
+    if (!st.ok()) {
+        LOG(WARNING) << "[ms-throttle] failed to create tick thread: " << st;
+    } else {
+        LOG(INFO) << "[ms-throttle] handler started: upgrade_cooldown="
+                  << config::ms_backpressure_upgrade_interval_ms
+                  << "ms, downgrade_interval=" << config::ms_backpressure_downgrade_interval_ms
+                  << "ms";
+    }
+}
+
+MSBackpressureHandler::~MSBackpressureHandler() {
+    _stop_latch.count_down();
+    if (_tick_thread) {
+        _tick_thread->join();
+    }
+}
+
+void MSBackpressureHandler::_tick_thread_callback() {
+    // Fixed tick interval: 1 second. Since 1 tick = 1 ms, advance by 1000 ticks each iteration.
+    constexpr int kTickIntervalMs = 1000;
+    while (!_stop_latch.wait_for(std::chrono::milliseconds(kTickIntervalMs))) {
+        _advance_time(kTickIntervalMs);
+    }
+}
+
+void MSBackpressureHandler::_advance_time(int ticks) {
+    if (!config::enable_ms_backpressure_handling) {
+        return;
+    }
+
+    // Advance coordinator time; if downgrade is triggered, handle it
+    if (_coordinator->tick(ticks)) {
+        LOG(INFO) << "[ms-throttle] triggering downgrade, upgrade_level="
+                  << _state_machine->upgrade_level();
+
+        auto actions = _state_machine->on_downgrade();
+        _apply_actions(actions);
+        _coordinator->set_has_pending_upgrades(_state_machine->upgrade_level() > 0);
+
+        g_backpressure_downgrade_count << 1;
+    }
+}
+
+bool MSBackpressureHandler::on_ms_busy() {

Review Comment:
   **Concurrency concern (Medium): TOCTOU race between `on_ms_busy()` and `_advance_time()`**
   
   Both `on_ms_busy()` (called from RPC threads) and `_advance_time()` (called from the tick thread) perform a multi-step sequence: coordinator decision → state machine mutation → `_apply_actions()` → coordinator feedback (`set_has_pending_upgrades`). Each individual call to `_coordinator` and `_state_machine` is internally synchronized, but the **sequence is not atomic**.
   
   Possible interleaving:
   1. Thread A (`on_ms_busy`): `_coordinator->report_ms_busy()` → returns `true` (upgrade)
   2. Thread B (`_advance_time`): `_coordinator->tick()` → returns `true` (downgrade)
   3. Thread B: `_state_machine->on_downgrade()` — may undo a previous upgrade
   4. Thread A: `_state_machine->on_upgrade(snapshot)` — applies upgrade
   5. Both set `_coordinator->set_has_pending_upgrades()` with potentially stale state
   
   The system is self-correcting (next `MS_BUSY` re-triggers), so this is **not a crash bug**, but can cause momentary incorrect throttle limits. Consider wrapping the entire upgrade/downgrade sequence in a handler-level mutex.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4044788647

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153365724

   # Cloud UT Coverage Report
   Increment line coverage `9.68% (3/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/e11f6697a68de51291f93879176dc7822361e9ed_e11f6697a68de51291f93879176dc7822361e9ed_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/e11f6697a68de51291f93879176dc7822361e9ed_e11f6697a68de51291f93879176dc7822361e9ed_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 78.64% (1797/2285) |
   | Line Coverage     | 64.42% (32335/50197) |
   | Region Coverage   | 65.27% (16203/24825) |
   | Branch Coverage   | 55.68% (8630/15498) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3885095915

   # BE UT Coverage Report
   Increment line coverage `61.31% (618/1008)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.83% (19544/36995) |
   | Line Coverage     | 36.31% (182073/501480) |
   | Region Coverage   | 32.72% (141422/432179) |
   | Branch Coverage   | 33.70% (61194/181565) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4152910540

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4167635039

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 35362 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 3e89cf3a7ca718c2d8a504573d9e974edf607060, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17648	4540	4325	4325
   q2	q3	10856	985	683	683
   q4	4809	501	345	345
   q5	7791	880	550	550
   q6	238	225	175	175
   q7	924	871	625	625
   q8	9511	1000	771	771
   q9	9051	6544	7610	6544
   q10	9016	4135	3728	3728
   q11	594	343	357	343
   q12	835	679	564	564
   q13	18099	4931	4013	4013
   q14	498	477	446	446
   q15	q16	581	551	461	461
   q17	977	963	761	761
   q18	6618	6054	5783	5783
   q19	1259	1335	886	886
   q20	1538	1411	1465	1411
   q21	4876	3008	2576	2576
   q22	476	396	372	372
   Total cold run time: 106195 ms
   Total hot run time: 35362 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	5391	5352	5262	5262
   q2	q3	1978	2404	2049	2049
   q4	1037	1408	907	907
   q5	4584	4897	4769	4769
   q6	231	210	157	157
   q7	8050	7885	7736	7736
   q8	3264	3672	3338	3338
   q9	28047	27844	27710	27710
   q10	4772	5114	4571	4571
   q11	1100	1028	991	991
   q12	623	705	538	538
   q13	4646	5248	4254	4254
   q14	507	503	476	476
   q15	q16	579	594	561	561
   q17	3948	4046	3910	3910
   q18	7580	7308	7045	7045
   q19	1233	1259	1232	1232
   q20	2361	2439	2282	2282
   q21	14066	13755	13613	13613
   q22	596	550	518	518
   Total cold run time: 94593 ms
   Total hot run time: 91919 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3884145907

   # FE UT Coverage Report
   Increment line coverage `  100.00% (10/10)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4171591724

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4169002824

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.57% (27283/37086) |
   | Line Coverage     | 57.15% (293568/513696) |
   | Region Coverage   | 54.24% (243841/449588) |
   | Branch Coverage   | 56.04% (105923/189017) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4047159865

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153057622

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168431577

   run cloudut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168434876

   run external


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4411713072

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #60344:
URL: https://github.com/apache/doris/pull/60344#discussion_r3008659593


##########
cloud/test/rate_limiter_test.cpp:
##########
@@ -228,7 +228,7 @@ TEST(RateLimiterTest, TestAdjustLimitInfluence1) {
         ASSERT_TRUE(meta_service->rate_limiter()->set_rate_limit(0));
         std::this_thread::sleep_for(std::chrono::seconds(1));
         mock_parallel_rpc(mock_get_cluster, meta_service.get(), mock_cloud_unique_id_0,
-                          MetaServiceCode::MAX_QPS_LIMIT, 1);
+                          MetaServiceCode::MS_TOO_BUSY, 1);

Review Comment:
   MS_TOO_BUSY and MAX_QPS_LIMIT are not equivalent



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153931882

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 169708 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit e11f6697a68de51291f93879176dc7822361e9ed, data reload: false
   
   query5	4352	652	520	520
   query6	347	229	234	229
   query7	4225	470	263	263
   query8	345	240	230	230
   query9	8735	2762	2738	2738
   query10	535	365	334	334
   query11	7016	5120	4860	4860
   query12	180	133	126	126
   query13	1279	477	353	353
   query14	5790	3679	3488	3488
   query14_1	2966	2877	2834	2834
   query15	204	201	179	179
   query16	991	461	459	459
   query17	906	730	620	620
   query18	2459	465	362	362
   query19	214	227	183	183
   query20	127	127	124	124
   query21	218	136	111	111
   query22	13302	14098	14231	14098
   query23	17162	16565	16129	16129
   query23_1	16433	16142	15890	15890
   query24	7176	1631	1220	1220
   query24_1	1239	1253	1251	1251
   query25	617	487	410	410
   query26	1240	273	154	154
   query27	2764	485	299	299
   query28	4523	1850	1855	1850
   query29	809	562	478	478
   query30	289	226	195	195
   query31	1013	958	888	888
   query32	88	72	71	71
   query33	536	342	283	283
   query34	893	868	505	505
   query35	637	687	606	606
   query36	1085	1126	993	993
   query37	141	97	82	82
   query38	2959	2943	2898	2898
   query39	850	834	808	808
   query39_1	792	797	807	797
   query40	234	157	137	137
   query41	63	64	61	61
   query42	260	256	255	255
   query43	243	255	227	227
   query44	
   query45	199	190	180	180
   query46	942	989	603	603
   query47	2135	2184	2073	2073
   query48	321	321	230	230
   query49	639	482	396	396
   query50	711	303	212	212
   query51	4090	4098	4057	4057
   query52	263	270	257	257
   query53	289	337	289	289
   query54	305	284	279	279
   query55	94	88	82	82
   query56	300	319	311	311
   query57	1915	1743	1764	1743
   query58	285	272	271	271
   query59	2823	2938	2737	2737
   query60	346	331	329	329
   query61	161	150	155	150
   query62	627	589	553	553
   query63	314	286	271	271
   query64	5010	1293	1024	1024
   query65	
   query66	1455	453	359	359
   query67	24289	24526	24274	24274
   query68	
   query69	409	318	287	287
   query70	933	941	922	922
   query71	347	305	306	305
   query72	2851	2474	2486	2474
   query73	552	558	328	328
   query74	9671	9614	9420	9420
   query75	2887	2768	2473	2473
   query76	2283	1044	663	663
   query77	369	378	312	312
   query78	10961	11132	10516	10516
   query79	1700	763	564	564
   query80	1360	636	541	541
   query81	561	257	223	223
   query82	987	157	121	121
   query83	340	275	238	238
   query84	307	115	109	109
   query85	951	490	449	449
   query86	402	309	307	307
   query87	3202	3120	3019	3019
   query88	3589	2669	2661	2661
   query89	417	383	346	346
   query90	2043	187	176	176
   query91	170	171	139	139
   query92	81	78	72	72
   query93	1029	879	501	501
   query94	655	328	298	298
   query95	612	406	332	332
   query96	656	524	231	231
   query97	2441	2455	2380	2380
   query98	235	220	217	217
   query99	1008	990	911	911
   Total cold run time: 252156 ms
   Total hot run time: 169708 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3883975731

   # Cloud UT Coverage Report
   Increment line coverage `9.68% (3/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 79.29% (1796/2265) |
   | Line Coverage     | 64.86% (32023/49369) |
   | Region Coverage   | 65.56% (15992/24394) |
   | Branch Coverage   | 56.07% (8505/15168) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4047642968

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3c9a97022829d3651babcc1963eeafe0f5bfb675_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3c9a97022829d3651babcc1963eeafe0f5bfb675_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.36% (26936/36718) |
   | Line Coverage     | 56.66% (287410/507252) |
   | Region Coverage   | 53.86% (239237/444188) |
   | Branch Coverage   | 55.59% (103323/185867) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3883842469

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168416344

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.57% (27286/37086) |
   | Line Coverage     | 57.14% (293528/513696) |
   | Region Coverage   | 54.21% (243731/449588) |
   | Branch Coverage   | 56.02% (105890/189017) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4171592095

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168158237

   # BE UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.99% (20064/37867) |
   | Line Coverage     | 36.54% (188291/515240) |
   | Region Coverage   | 32.82% (146214/445472) |
   | Branch Coverage   | 33.94% (63963/188451) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153871462

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 26863 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit e11f6697a68de51291f93879176dc7822361e9ed, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17670	4559	4435	4435
   q2	q3	10729	805	529	529
   q4	4735	363	251	251
   q5	8111	1227	997	997
   q6	224	174	144	144
   q7	794	860	683	683
   q8	10556	1515	1361	1361
   q9	6542	4786	4780	4780
   q10	6294	1934	1677	1677
   q11	499	255	239	239
   q12	728	591	459	459
   q13	18068	2822	1960	1960
   q14	238	239	209	209
   q15	q16	730	748	669	669
   q17	736	852	460	460
   q18	6221	5397	5185	5185
   q19	1117	987	621	621
   q20	553	479	377	377
   q21	4350	2133	1535	1535
   q22	370	345	292	292
   Total cold run time: 99265 ms
   Total hot run time: 26863 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4745	4663	4577	4577
   q2	q3	3864	4349	3859	3859
   q4	898	1211	811	811
   q5	4047	4457	4351	4351
   q6	180	171	142	142
   q7	1803	1636	1534	1534
   q8	2528	2751	2591	2591
   q9	7843	7636	7461	7461
   q10	3823	3978	3613	3613
   q11	508	442	442	442
   q12	592	618	446	446
   q13	2465	3160	2124	2124
   q14	289	312	287	287
   q15	q16	758	822	718	718
   q17	1231	1407	1376	1376
   q18	7279	6960	6735	6735
   q19	908	955	933	933
   q20	2105	2146	2020	2020
   q21	3970	3465	3360	3360
   q22	464	429	386	386
   Total cold run time: 50300 ms
   Total hot run time: 47766 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60344:
URL: https://github.com/apache/doris/pull/60344#discussion_r3008312337


##########
be/src/cloud/cloud_throttle_state_machine.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_throttle_state_machine.h"
+
+#include <glog/logging.h>
+
+#include <algorithm>
+
+namespace doris::cloud {
+
+// Display names for LoadRelatedRpc types
+static constexpr std::string_view LOAD_RELATED_RPC_NAMES[] = {
+        "prepare_rowset", "commit_rowset", "update_tmp_rowset", "update_packed_file_info",
+        "update_delete_bitmap"};
+
+std::string_view load_related_rpc_name(LoadRelatedRpc rpc) {
+    size_t idx = static_cast<size_t>(rpc);
+    if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+        return LOAD_RELATED_RPC_NAMES[idx];
+    }
+    return "unknown";
+}
+
+// ============== RpcThrottleStateMachine ==============
+
+RpcThrottleStateMachine::RpcThrottleStateMachine(RpcThrottleParams params) : _params(params) {
+    LOG(INFO) << "[ms-throttle] state machine initialized: top_k=" << params.top_k
+              << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+void RpcThrottleStateMachine::update_params(RpcThrottleParams params) {
+    std::lock_guard lock(_mtx);
+    _params = params;
+    LOG(INFO) << "[ms-throttle] state machine params updated: top_k=" << params.top_k
+              << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_upgrade(
+        const std::vector<RpcQpsSnapshot>& qps_snapshot) {
+    std::lock_guard lock(_mtx);
+
+    UpgradeRecord record;
+    std::vector<RpcThrottleAction> actions;
+
+    double ratio = _params.ratio;
+    double floor_qps = _params.floor_qps;
+
+    // Caller is responsible for providing top-k snapshot per RPC type.
+    // State machine simply applies throttling to every entry in the snapshot.
+    for (const auto& snapshot : qps_snapshot) {
+        auto key = std::make_pair(snapshot.rpc_type, snapshot.table_id);
+
+        double old_limit = 0.0;
+        auto limit_it = _current_limits.find(key);
+        if (limit_it != _current_limits.end()) {
+            old_limit = limit_it->second;
+        }
+
+        double new_limit;
+        if (old_limit > 0) {
+            // Already has a limit, reduce it further
+            new_limit = old_limit * ratio;
+        } else {
+            // No limit yet, set based on current QPS
+            new_limit = snapshot.current_qps * ratio;
+        }
+
+        // Apply floor
+        new_limit = std::max(new_limit, floor_qps);
+
+        // Only apply if it's actually limiting
+        if (new_limit < snapshot.current_qps || old_limit > 0) {
+            RpcThrottleAction action {
+                    .type = RpcThrottleAction::Type::SET_LIMIT,
+                    .rpc_type = snapshot.rpc_type,
+                    .table_id = snapshot.table_id,
+                    .qps_limit = new_limit,
+            };
+            actions.push_back(action);
+            record.changes[key] = {old_limit, new_limit};
+            _current_limits[key] = new_limit;
+
+            LOG(INFO) << "[ms-throttle] upgrade: rpc=" << load_related_rpc_name(snapshot.rpc_type)
+                      << ", table_id=" << snapshot.table_id
+                      << ", current_qps=" << snapshot.current_qps << ", old_limit=" << old_limit
+                      << ", new_limit=" << new_limit;
+        }
+    }
+
+    if (!record.changes.empty()) {
+        _upgrade_history.push_back(std::move(record));
+    }
+
+    LOG(INFO) << "[ms-throttle] on_upgrade done: actions=" << actions.size()
+              << ", upgrade_level=" << _upgrade_history.size()
+              << ", snapshot_size=" << qps_snapshot.size();
+
+    return actions;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_downgrade() {
+    std::lock_guard lock(_mtx);
+
+    std::vector<RpcThrottleAction> actions;
+
+    if (_upgrade_history.empty()) {
+        LOG(INFO) << "[ms-throttle] on_downgrade skipped: no upgrade history";
+        return actions;
+    }
+
+    // Undo the most recent upgrade
+    const auto& record = _upgrade_history.back();
+
+    for (const auto& [key, limits] : record.changes) {
+        const auto& [rpc_type, table_id] = key;
+        double old_limit = limits.first;
+
+        if (old_limit > 0) {
+            // Restore the previous limit
+            RpcThrottleAction action {
+                    .type = RpcThrottleAction::Type::SET_LIMIT,
+                    .rpc_type = rpc_type,
+                    .table_id = table_id,
+                    .qps_limit = old_limit,
+            };
+
+            actions.push_back(action);
+            _current_limits[key] = old_limit;
+
+            LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+                      << ", table_id=" << table_id << ", restored_limit=" << old_limit;
+        } else {
+            // No previous limit, remove it entirely
+            RpcThrottleAction action {
+                    .type = RpcThrottleAction::Type::REMOVE_LIMIT,
+                    .rpc_type = rpc_type,
+                    .table_id = table_id,
+            };
+
+            actions.push_back(action);
+            _current_limits.erase(key);
+
+            LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+                      << ", table_id=" << table_id << ", removed limit";
+        }
+    }
+
+    _upgrade_history.pop_back();
+
+    LOG(INFO) << "[ms-throttle] on_downgrade done: actions=" << actions.size()
+              << ", upgrade_level=" << _upgrade_history.size();
+
+    return actions;
+}
+
+size_t RpcThrottleStateMachine::upgrade_level() const {
+    std::lock_guard lock(_mtx);
+    return _upgrade_history.size();
+}
+
+double RpcThrottleStateMachine::get_current_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+    std::lock_guard lock(_mtx);
+    auto it = _current_limits.find({rpc_type, table_id});
+    if (it != _current_limits.end()) {
+        return it->second;
+    }
+    return 0.0;
+}
+
+RpcThrottleParams RpcThrottleStateMachine::get_params() const {
+    std::lock_guard lock(_mtx);
+    return _params;
+}
+
+// ============== RpcThrottleCoordinator ==============
+
+RpcThrottleCoordinator::RpcThrottleCoordinator(ThrottleCoordinatorParams params) : _params(params) {
+    LOG(INFO) << "[ms-throttle] coordinator initialized: upgrade_cooldown_ticks="
+              << params.upgrade_cooldown_ticks
+              << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+void RpcThrottleCoordinator::update_params(ThrottleCoordinatorParams params) {
+    std::lock_guard lock(_mtx);
+    _params = params;
+    LOG(INFO) << "[ms-throttle] coordinator params updated: upgrade_cooldown_ticks="
+              << params.upgrade_cooldown_ticks
+              << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+bool RpcThrottleCoordinator::report_ms_busy() {
+    std::lock_guard lock(_mtx);
+
+    // Reset tick counter since last MS_BUSY
+    _ticks_since_last_ms_busy = 0;
+
+    // Check if cooldown has passed
+    if (_ticks_since_last_upgrade == -1 ||
+        _ticks_since_last_upgrade >= _params.upgrade_cooldown_ticks) {
+        // Reset upgrade counter
+        _ticks_since_last_upgrade = 0;
+        _has_pending_upgrades = true;
+
+        LOG(INFO) << "[ms-throttle] upgrade triggered: ticks_since_last_upgrade="
+                  << _ticks_since_last_upgrade << ", cooldown=" << _params.upgrade_cooldown_ticks;
+        return true; // Should trigger upgrade
+    }
+    return false; // Cooling down
+}
+
+bool RpcThrottleCoordinator::tick(int ticks) {
+    std::lock_guard lock(_mtx);
+
+    // Increment tick counters
+    if (_ticks_since_last_ms_busy >= 0) {
+        _ticks_since_last_ms_busy += ticks;
+    }
+    if (_ticks_since_last_upgrade >= 0) {
+        _ticks_since_last_upgrade += ticks;
+    }
+
+    // Check if downgrade should be triggered
+    if (_has_pending_upgrades && _ticks_since_last_ms_busy >= _params.downgrade_after_ticks) {
+        // Reset for next downgrade cycle
+        _ticks_since_last_ms_busy = 0;
+
+        LOG(INFO) << "[ms-throttle] downgrade triggered: ticks_since_last_ms_busy="

Review Comment:
   **Bug: Log message always prints `downgrade_after_ticks` instead of actual elapsed ticks.**
   
   At this point, `_ticks_since_last_ms_busy` has already been reset to `0` on line 240. So the expression `_ticks_since_last_ms_busy + _params.downgrade_after_ticks` always evaluates to `0 + downgrade_after_ticks = downgrade_after_ticks`, not the actual ticks that elapsed.
   
   Fix: capture the value before resetting:
   ```cpp
   auto actual_ticks = _ticks_since_last_ms_busy;
   _ticks_since_last_ms_busy = 0;
   LOG(INFO) << "[ms-throttle] downgrade triggered: ticks_since_last_ms_busy="
             << actual_ticks << ", threshold=" << _params.downgrade_after_ticks;
   ```



##########
be/src/cloud/cloud_throttle_state_machine.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_throttle_state_machine.h"
+
+#include <glog/logging.h>
+
+#include <algorithm>
+
+namespace doris::cloud {
+
+// Display names for LoadRelatedRpc types
+static constexpr std::string_view LOAD_RELATED_RPC_NAMES[] = {
+        "prepare_rowset", "commit_rowset", "update_tmp_rowset", "update_packed_file_info",
+        "update_delete_bitmap"};
+
+std::string_view load_related_rpc_name(LoadRelatedRpc rpc) {
+    size_t idx = static_cast<size_t>(rpc);
+    if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+        return LOAD_RELATED_RPC_NAMES[idx];
+    }
+    return "unknown";
+}
+
+// ============== RpcThrottleStateMachine ==============
+
+RpcThrottleStateMachine::RpcThrottleStateMachine(RpcThrottleParams params) : _params(params) {
+    LOG(INFO) << "[ms-throttle] state machine initialized: top_k=" << params.top_k
+              << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+void RpcThrottleStateMachine::update_params(RpcThrottleParams params) {
+    std::lock_guard lock(_mtx);
+    _params = params;
+    LOG(INFO) << "[ms-throttle] state machine params updated: top_k=" << params.top_k
+              << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_upgrade(
+        const std::vector<RpcQpsSnapshot>& qps_snapshot) {
+    std::lock_guard lock(_mtx);
+
+    UpgradeRecord record;
+    std::vector<RpcThrottleAction> actions;
+
+    double ratio = _params.ratio;
+    double floor_qps = _params.floor_qps;
+
+    // Caller is responsible for providing top-k snapshot per RPC type.
+    // State machine simply applies throttling to every entry in the snapshot.
+    for (const auto& snapshot : qps_snapshot) {
+        auto key = std::make_pair(snapshot.rpc_type, snapshot.table_id);
+
+        double old_limit = 0.0;
+        auto limit_it = _current_limits.find(key);
+        if (limit_it != _current_limits.end()) {
+            old_limit = limit_it->second;
+        }
+
+        double new_limit;
+        if (old_limit > 0) {
+            // Already has a limit, reduce it further
+            new_limit = old_limit * ratio;
+        } else {
+            // No limit yet, set based on current QPS
+            new_limit = snapshot.current_qps * ratio;
+        }
+
+        // Apply floor
+        new_limit = std::max(new_limit, floor_qps);
+
+        // Only apply if it's actually limiting
+        if (new_limit < snapshot.current_qps || old_limit > 0) {
+            RpcThrottleAction action {
+                    .type = RpcThrottleAction::Type::SET_LIMIT,
+                    .rpc_type = snapshot.rpc_type,
+                    .table_id = snapshot.table_id,
+                    .qps_limit = new_limit,
+            };
+            actions.push_back(action);
+            record.changes[key] = {old_limit, new_limit};
+            _current_limits[key] = new_limit;
+
+            LOG(INFO) << "[ms-throttle] upgrade: rpc=" << load_related_rpc_name(snapshot.rpc_type)
+                      << ", table_id=" << snapshot.table_id
+                      << ", current_qps=" << snapshot.current_qps << ", old_limit=" << old_limit
+                      << ", new_limit=" << new_limit;
+        }
+    }
+
+    if (!record.changes.empty()) {
+        _upgrade_history.push_back(std::move(record));
+    }
+
+    LOG(INFO) << "[ms-throttle] on_upgrade done: actions=" << actions.size()
+              << ", upgrade_level=" << _upgrade_history.size()
+              << ", snapshot_size=" << qps_snapshot.size();
+
+    return actions;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_downgrade() {
+    std::lock_guard lock(_mtx);
+
+    std::vector<RpcThrottleAction> actions;
+
+    if (_upgrade_history.empty()) {
+        LOG(INFO) << "[ms-throttle] on_downgrade skipped: no upgrade history";
+        return actions;
+    }
+
+    // Undo the most recent upgrade
+    const auto& record = _upgrade_history.back();
+
+    for (const auto& [key, limits] : record.changes) {
+        const auto& [rpc_type, table_id] = key;
+        double old_limit = limits.first;
+
+        if (old_limit > 0) {
+            // Restore the previous limit
+            RpcThrottleAction action {
+                    .type = RpcThrottleAction::Type::SET_LIMIT,
+                    .rpc_type = rpc_type,
+                    .table_id = table_id,
+                    .qps_limit = old_limit,
+            };
+
+            actions.push_back(action);
+            _current_limits[key] = old_limit;
+
+            LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+                      << ", table_id=" << table_id << ", restored_limit=" << old_limit;
+        } else {
+            // No previous limit, remove it entirely
+            RpcThrottleAction action {
+                    .type = RpcThrottleAction::Type::REMOVE_LIMIT,
+                    .rpc_type = rpc_type,
+                    .table_id = table_id,
+            };
+
+            actions.push_back(action);
+            _current_limits.erase(key);
+
+            LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+                      << ", table_id=" << table_id << ", removed limit";
+        }
+    }
+
+    _upgrade_history.pop_back();
+
+    LOG(INFO) << "[ms-throttle] on_downgrade done: actions=" << actions.size()
+              << ", upgrade_level=" << _upgrade_history.size();
+
+    return actions;
+}
+
+size_t RpcThrottleStateMachine::upgrade_level() const {
+    std::lock_guard lock(_mtx);
+    return _upgrade_history.size();
+}
+
+double RpcThrottleStateMachine::get_current_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+    std::lock_guard lock(_mtx);
+    auto it = _current_limits.find({rpc_type, table_id});
+    if (it != _current_limits.end()) {
+        return it->second;
+    }
+    return 0.0;
+}
+
+RpcThrottleParams RpcThrottleStateMachine::get_params() const {
+    std::lock_guard lock(_mtx);
+    return _params;
+}
+
+// ============== RpcThrottleCoordinator ==============
+
+RpcThrottleCoordinator::RpcThrottleCoordinator(ThrottleCoordinatorParams params) : _params(params) {
+    LOG(INFO) << "[ms-throttle] coordinator initialized: upgrade_cooldown_ticks="
+              << params.upgrade_cooldown_ticks
+              << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+void RpcThrottleCoordinator::update_params(ThrottleCoordinatorParams params) {
+    std::lock_guard lock(_mtx);
+    _params = params;
+    LOG(INFO) << "[ms-throttle] coordinator params updated: upgrade_cooldown_ticks="
+              << params.upgrade_cooldown_ticks
+              << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+bool RpcThrottleCoordinator::report_ms_busy() {
+    std::lock_guard lock(_mtx);
+
+    // Reset tick counter since last MS_BUSY
+    _ticks_since_last_ms_busy = 0;
+
+    // Check if cooldown has passed
+    if (_ticks_since_last_upgrade == -1 ||
+        _ticks_since_last_upgrade >= _params.upgrade_cooldown_ticks) {
+        // Reset upgrade counter
+        _ticks_since_last_upgrade = 0;
+        _has_pending_upgrades = true;
+
+        LOG(INFO) << "[ms-throttle] upgrade triggered: ticks_since_last_upgrade="

Review Comment:
   **Bug: Log message always prints 0 for `ticks_since_last_upgrade`.**
   
   `_ticks_since_last_upgrade` was just set to `0` on line 216, so this log always prints `0` instead of the actual number of ticks since the last upgrade (which would show how long the cooldown actually lasted).
   
   Fix: capture before resetting:
   ```cpp
   auto actual_ticks = _ticks_since_last_upgrade;
   _ticks_since_last_upgrade = 0;
   _has_pending_upgrades = true;
   LOG(INFO) << "[ms-throttle] upgrade triggered: ticks_since_last_upgrade="
             << actual_ticks << ", cooldown=" << _params.upgrade_cooldown_ticks;
   ```



##########
be/src/cloud/config.cpp:
##########
@@ -173,5 +173,49 @@ DEFINE_mInt64(cache_read_from_peer_expired_seconds, "-1");
 DEFINE_mBool(enable_file_cache_write_base_compaction_index_only, "false");
 DEFINE_mBool(enable_file_cache_write_cumu_compaction_index_only, "false");
 
+// MS RPC rate limiting config
+DEFINE_mBool(enable_ms_rpc_host_level_rate_limit, "true");
+
+// Per-RPC QPS limit configs (per CPU core)
+// QPS limit = config_value * num_cores
+// Set to 0 to disable rate limiting for a specific RPC
+// Set to -1 to use ms_rpc_qps_default config value
+DEFINE_mInt32(ms_rpc_qps_default, "100");
+DEFINE_mInt32(ms_rpc_qps_get_tablet_meta, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_prepare_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_commit_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_update_tmp_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_commit_txn, "-1");
+DEFINE_mInt32(ms_rpc_qps_abort_txn, "-1");
+DEFINE_mInt32(ms_rpc_qps_precommit_txn, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_obj_store_info, "-1");
+DEFINE_mInt32(ms_rpc_qps_start_tablet_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_finish_tablet_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_delete_bitmap, "-1");
+DEFINE_mInt32(ms_rpc_qps_update_delete_bitmap, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_delete_bitmap_update_lock, "-1");
+DEFINE_mInt32(ms_rpc_qps_remove_delete_bitmap_update_lock, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_instance, "-1");
+DEFINE_mInt32(ms_rpc_qps_prepare_restore_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_commit_restore_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_finish_restore_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_list_snapshots, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_cluster_status, "-1");
+DEFINE_mInt32(ms_rpc_qps_update_packed_file_info, "-1");
+
+// Table-level backpressure handling config
+DEFINE_mBool(enable_ms_backpressure_handling, "true");

Review Comment:
   **Doc mismatch**: The PR description table says `enable_ms_backpressure_handling` defaults to `false`, but here it defaults to `"true"`. Please align the PR description with the code, or change the default if `false` was intended.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4167217808

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4046262629

   # BE UT Coverage Report
   Increment line coverage `61.17% (619/1012)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/3c9a97022829d3651babcc1963eeafe0f5bfb675_3c9a97022829d3651babcc1963eeafe0f5bfb675/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/3c9a97022829d3651babcc1963eeafe0f5bfb675_3c9a97022829d3651babcc1963eeafe0f5bfb675/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.68% (19749/37486) |
   | Line Coverage     | 36.23% (184114/508156) |
   | Region Coverage   | 32.38% (142321/439594) |
   | Branch Coverage   | 33.51% (62026/185103) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4384935506

   run check_coverage_fe


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3884403194

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 30385 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17679	4467	4272	4272
   q2	2039	390	244	244
   q3	10109	1308	749	749
   q4	10207	793	319	319
   q5	7551	2209	1958	1958
   q6	205	184	150	150
   q7	882	748	619	619
   q8	9263	1420	1155	1155
   q9	4769	4700	4617	4617
   q10	6827	1937	1546	1546
   q11	494	265	237	237
   q12	345	383	229	229
   q13	17782	4100	3220	3220
   q14	231	233	223	223
   q15	906	821	818	818
   q16	701	679	604	604
   q17	713	816	547	547
   q18	7109	5985	5815	5815
   q19	1106	1011	627	627
   q20	513	503	393	393
   q21	2575	1838	1794	1794
   q22	327	289	249	249
   Total cold run time: 102333 ms
   Total hot run time: 30385 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4383	4324	4319	4319
   q2	280	336	259	259
   q3	2125	2720	2212	2212
   q4	1368	1746	1307	1307
   q5	4336	4218	4183	4183
   q6	220	193	137	137
   q7	1879	1830	1702	1702
   q8	2673	2631	2489	2489
   q9	7645	7429	7470	7429
   q10	2810	3278	2678	2678
   q11	501	439	433	433
   q12	682	728	644	644
   q13	3895	4764	3545	3545
   q14	308	348	302	302
   q15	849	810	807	807
   q16	698	740	678	678
   q17	1146	1339	1393	1339
   q18	8283	7969	8028	7969
   q19	887	884	859	859
   q20	2104	2189	2065	2065
   q21	4771	4677	4393	4393
   q22	527	469	485	469
   Total cold run time: 52370 ms
   Total hot run time: 50218 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3884489766

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 28.3 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa, data reload: false
   
   query1	0.05	0.04	0.04
   query2	0.10	0.05	0.05
   query3	0.25	0.08	0.08
   query4	1.61	0.12	0.11
   query5	0.27	0.26	0.25
   query6	1.16	0.68	0.66
   query7	0.03	0.03	0.03
   query8	0.06	0.04	0.04
   query9	0.56	0.52	0.49
   query10	0.55	0.54	0.54
   query11	0.14	0.10	0.10
   query12	0.14	0.10	0.10
   query13	0.64	0.61	0.62
   query14	1.06	1.06	1.05
   query15	0.88	0.86	0.87
   query16	0.41	0.39	0.40
   query17	1.07	1.05	1.14
   query18	0.22	0.22	0.21
   query19	2.10	2.02	2.01
   query20	0.02	0.02	0.01
   query21	15.39	0.26	0.15
   query22	5.43	0.06	0.05
   query23	16.17	0.28	0.11
   query24	1.49	0.34	0.23
   query25	0.09	0.07	0.08
   query26	0.15	0.14	0.15
   query27	0.11	0.09	0.05
   query28	3.59	1.17	0.97
   query29	12.56	3.95	3.19
   query30	0.27	0.14	0.12
   query31	2.81	0.65	0.41
   query32	3.25	0.58	0.50
   query33	3.24	3.25	3.27
   query34	16.04	5.41	4.73
   query35	4.80	4.80	4.80
   query36	0.65	0.50	0.48
   query37	0.12	0.07	0.07
   query38	0.08	0.04	0.05
   query39	0.05	0.03	0.03
   query40	0.22	0.16	0.14
   query41	0.09	0.03	0.03
   query42	0.05	0.03	0.03
   query43	0.05	0.04	0.03
   Total cold run time: 98.02 s
   Total hot run time: 28.3 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4393695651

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168636818

   # Cloud UT Coverage Report
   Increment line coverage `9.68% (3/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 78.46% (1799/2293) |
   | Line Coverage     | 64.18% (32322/50365) |
   | Region Coverage   | 65.01% (16215/24942) |
   | Branch Coverage   | 55.46% (8640/15580) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4169096965

   run external


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4169608003

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.57% (27285/37086) |
   | Line Coverage     | 57.15% (293553/513696) |
   | Region Coverage   | 54.26% (243944/449588) |
   | Branch Coverage   | 56.05% (105946/189017) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153134910

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #60344:
URL: https://github.com/apache/doris/pull/60344


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org