You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2026/01/29 06:55:29 UTC
[PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
bobhan1 opened a new pull request, #60344:
URL: https://github.com/apache/doris/pull/60344
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3815832759
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4388226608
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153568462
# FE UT Coverage Report
Increment line coverage ` 100.00% (10/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_e11f6697a68de51291f93879176dc7822361e9ed/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_e11f6697a68de51291f93879176dc7822361e9ed/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60344:
URL: https://github.com/apache/doris/pull/60344#discussion_r2925187722
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -423,6 +504,10 @@ Status retry_rpc(std::string_view op_name, const Request& req, Response* res,
} else if (res->status().code() == MetaServiceCode::INVALID_ARGUMENT) {
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>("failed to {}: {}", op_name,
res->status().msg());
+ } else if (res->status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
+ // TODO(bobhan1): change to correct error code
+ rate_limit_ctx.backpressure_handler->on_ms_busy();
Review Comment:
**Bug (Critical): Null pointer dereference**
`rate_limit_ctx.backpressure_handler` can be `nullptr`. The `RpcRateLimitCtx` struct defaults `backpressure_handler` to `nullptr`, and `get_cluster_status` (line ~2516) passes `{.host_limiters = host_level_ms_rpc_rate_limiters_}` without setting `.backpressure_handler`. If MS returns `MAX_QPS_LIMIT` for that RPC, this will crash.
Both `apply_rate_limit()` (line 418) and `record_rpc_qps()` (line 446) correctly null-check `ctx.backpressure_handler` before use. The same guard is needed here.
Suggested fix:
```cpp
} else if (res->status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
if (rate_limit_ctx.backpressure_handler) {
rate_limit_ctx.backpressure_handler->on_ms_busy();
}
// MS_BUSY should also be retried
```
##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -631,6 +727,12 @@ Status CloudMetaMgr::sync_tablet_rowsets_unlocked(CloudTablet* tablet,
return Status::NotFound("failed to get rowset meta: {}, {}", resp.status().msg(),
tablet_info);
}
+ if (resp.status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
+ // TODO(bobhan1): change to correct error code
+ ms_backpressure_handler_->on_ms_busy();
Review Comment:
**Bug (Critical): Null pointer dereference + Infinite loop**
Two issues here:
**1. Null deref:** `ms_backpressure_handler_` is initialized to `nullptr` (see `cloud_meta_mgr.h`) and only set via `set_ms_backpressure_handler()`. If this setter was never called, or called with null, a `MAX_QPS_LIMIT` response will crash. Line 698 correctly null-checks `host_level_ms_rpc_rate_limiters_` in the same function — the backpressure handler should get the same treatment.
**2. Infinite busy-loop:** The `continue` on line 734 skips incrementing `tried` and has no backoff/sleep. If MS keeps returning `MAX_QPS_LIMIT`, this is an unbounded busy-loop consuming 100% CPU. Contrast with `retry_rpc()` which increments `retry_times` and applies exponential backoff.
Suggested fix:
```cpp
if (resp.status().code() == MetaServiceCode::MAX_QPS_LIMIT) {
if (ms_backpressure_handler_) {
ms_backpressure_handler_->on_ms_busy();
}
++tried; // Count against retry limit
// Apply backoff similar to retry_rpc
bthread_usleep(std::min(1000 * (1 << tried), 60000) * 1000ULL);
continue;
}
```
##########
be/src/cloud/cloud_ms_backpressure_handler.cpp:
##########
@@ -0,0 +1,517 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_ms_backpressure_handler.h"
+
+#include <fmt/format.h>
+#include <glog/logging.h>
+
+#include <algorithm>
+#include <cmath>
+#include <queue>
+#include <thread>
+
+#include "cloud/config.h"
+#include "common/status.h"
+#include "util/thread.h"
+
+namespace doris::cloud {
+
+// Global bvar metrics
+bvar::Adder<uint64_t> g_backpressure_upgrade_count("ms_rpc_backpressure_upgrade_count");
+bvar::Window<bvar::Adder<uint64_t>> g_backpressure_upgrade_60s("ms_rpc_backpressure_upgrade_60s",
+ &g_backpressure_upgrade_count, 60);
+bvar::Adder<uint64_t> g_backpressure_downgrade_count("ms_rpc_backpressure_downgrade_count");
+bvar::Window<bvar::Adder<uint64_t>> g_backpressure_downgrade_60s(
+ "ms_rpc_backpressure_downgrade_60s", &g_backpressure_downgrade_count, 60);
+bvar::LatencyRecorder g_throttle_wait_prepare_rowset(
+ "ms_rpc_backpressure_throttle_wait_prepare_rowset");
+bvar::LatencyRecorder g_throttle_wait_commit_rowset(
+ "ms_rpc_backpressure_throttle_wait_commit_rowset");
+bvar::LatencyRecorder g_throttle_wait_update_tmp_rowset(
+ "ms_rpc_backpressure_throttle_wait_update_tmp_rowset");
+bvar::LatencyRecorder g_throttle_wait_update_packed_file_info(
+ "ms_rpc_backpressure_throttle_wait_update_packed_file_info");
+bvar::LatencyRecorder g_throttle_wait_update_delete_bitmap(
+ "ms_rpc_backpressure_throttle_wait_update_delete_bitmap");
+bvar::Adder<uint64_t> g_ms_busy_count("ms_rpc_backpressure_ms_busy_count");
+bvar::Window<bvar::Adder<uint64_t>> g_ms_busy_60s("ms_rpc_backpressure_ms_busy_60s",
+ &g_ms_busy_count, 60);
+
+static bvar::LatencyRecorder* s_throttle_wait_recorders[] = {
+ &g_throttle_wait_prepare_rowset, &g_throttle_wait_commit_rowset,
+ &g_throttle_wait_update_tmp_rowset, &g_throttle_wait_update_packed_file_info,
+ &g_throttle_wait_update_delete_bitmap,
+};
+
+bvar::LatencyRecorder* get_throttle_wait_recorder(LoadRelatedRpc rpc) {
+ size_t idx = static_cast<size_t>(rpc);
+ if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ return nullptr;
+ }
+ return s_throttle_wait_recorders[idx];
+}
+
+// ============== StrictQpsLimiter ==============
+
+StrictQpsLimiter::StrictQpsLimiter(double qps) {
+ if (qps <= 0) {
+ qps = 1.0;
+ }
+ _interval_ns = static_cast<int64_t>(1e9 / qps);
+ _next_allowed_time = Clock::now();
+}
+
+StrictQpsLimiter::Clock::time_point StrictQpsLimiter::reserve() {
+ std::lock_guard lock(_mtx);
+ auto now = Clock::now();
+ if (_next_allowed_time <= now) {
+ _next_allowed_time = now + std::chrono::nanoseconds(_interval_ns);
+ return now;
+ }
+ auto result = _next_allowed_time;
+ _next_allowed_time += std::chrono::nanoseconds(_interval_ns);
+ return result;
+}
+
+void StrictQpsLimiter::update_qps(double new_qps) {
+ if (new_qps <= 0) {
+ new_qps = 1.0;
+ }
+ std::lock_guard lock(_mtx);
+ _interval_ns = static_cast<int64_t>(1e9 / new_qps);
+}
+
+double StrictQpsLimiter::get_qps() const {
+ std::lock_guard lock(_mtx);
+ if (_interval_ns <= 0) {
+ return 0;
+ }
+ return 1e9 / _interval_ns;
+}
+
+// ============== TableRpcQpsCounter ==============
+
+TableRpcQpsCounter::TableRpcQpsCounter(int64_t table_id, LoadRelatedRpc rpc_type, int window_sec)
+ : _table_id(table_id), _rpc_type(rpc_type) {
+ _counter = std::make_unique<bvar::Adder<int64_t>>();
+ _counter->hide();
+ _qps = std::make_unique<bvar::PerSecond<bvar::Adder<int64_t>>>(_counter.get(), window_sec);
+ _qps->hide();
+}
+
+void TableRpcQpsCounter::increment() {
+ (*_counter) << 1;
+}
+
+double TableRpcQpsCounter::get_qps() const {
+ return _qps->get_value();
+}
+
+// ============== TableRpcQpsRegistry ==============
+
+TableRpcQpsRegistry::TableRpcQpsRegistry() = default;
+
+void TableRpcQpsRegistry::record(LoadRelatedRpc rpc_type, int64_t table_id) {
+ auto* counter = get_or_create_counter(rpc_type, table_id);
+ if (counter) {
+ counter->increment();
+ }
+}
+
+TableRpcQpsCounter* TableRpcQpsRegistry::get_or_create_counter(LoadRelatedRpc rpc_type,
+ int64_t table_id) {
+ size_t idx = static_cast<size_t>(rpc_type);
+ if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ return nullptr;
+ }
+
+ {
+ std::shared_lock lock(_mutex);
+ auto it = _counters[idx].find(table_id);
+ if (it != _counters[idx].end()) {
+ return it->second.get();
+ }
+ }
+
+ std::unique_lock lock(_mutex);
+ // Double check after acquiring exclusive lock
+ auto it = _counters[idx].find(table_id);
+ if (it != _counters[idx].end()) {
+ return it->second.get();
+ }
+
+ auto counter = std::make_unique<TableRpcQpsCounter>(table_id, rpc_type,
+ config::ms_rpc_table_qps_window_sec);
+ auto* ptr = counter.get();
+ _counters[idx][table_id] = std::move(counter);
+ return ptr;
+}
+
+std::vector<std::pair<int64_t, double>> TableRpcQpsRegistry::get_top_k_tables(
+ LoadRelatedRpc rpc_type, int k) const {
+ size_t idx = static_cast<size_t>(rpc_type);
+ if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT) || k <= 0) {
+ return {};
+ }
+
+ // Use a min-heap of size k to find top-k without allocating a vector for all tables.
+ // The heap top is the smallest among the k largest elements seen so far.
+ using Entry = std::pair<int64_t, double>; // (table_id, qps)
+ auto min_cmp = [](const Entry& a, const Entry& b) { return a.second > b.second; };
+ std::priority_queue<Entry, std::vector<Entry>, decltype(min_cmp)> min_heap(min_cmp);
+
+ {
+ std::shared_lock lock(_mutex);
+ for (const auto& [table_id, counter] : _counters[idx]) {
+ double qps = counter->get_qps();
+ if (qps > 0) {
+ if (static_cast<int>(min_heap.size()) < k) {
+ min_heap.push({table_id, qps});
+ } else if (qps > min_heap.top().second) {
+ min_heap.pop();
+ min_heap.push({table_id, qps});
+ }
+ }
+ }
+ }
+
+ // Extract results from heap (comes out in ascending order, reverse to descending)
+ std::vector<Entry> result;
+ result.reserve(min_heap.size());
+ while (!min_heap.empty()) {
+ result.push_back(min_heap.top());
+ min_heap.pop();
+ }
+ std::reverse(result.begin(), result.end());
+
+ return result;
+}
+
+double TableRpcQpsRegistry::get_qps(LoadRelatedRpc rpc_type, int64_t table_id) const {
+ size_t idx = static_cast<size_t>(rpc_type);
+ if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ return 0;
+ }
+
+ std::shared_lock lock(_mutex);
+ auto it = _counters[idx].find(table_id);
+ if (it != _counters[idx].end()) {
+ return it->second->get_qps();
+ }
+ return 0;
+}
+
+void TableRpcQpsRegistry::cleanup_inactive_tables() {
+ std::unique_lock lock(_mutex);
+
+ for (size_t idx = 0; idx < static_cast<size_t>(LoadRelatedRpc::COUNT); ++idx) {
+ auto& counter_map = _counters[idx];
+ for (auto it = counter_map.begin(); it != counter_map.end();) {
+ // Remove counters with zero QPS for a long time
+ if (it->second->get_qps() < 0.01) {
+ it = counter_map.erase(it);
+ } else {
+ ++it;
+ }
+ }
+ }
+}
+
+// ============== TableRpcThrottler ==============
+
+TableRpcThrottler::TableRpcThrottler() {
+ // Initialize bvar for throttled table counts
+ for (size_t i = 0; i < static_cast<size_t>(LoadRelatedRpc::COUNT); ++i) {
+ std::string bvar_name = fmt::format("ms_rpc_backpressure_throttled_tables_{}",
+ load_related_rpc_name(static_cast<LoadRelatedRpc>(i)));
+ _throttled_table_counts[i] = std::make_unique<bvar::Status<size_t>>(bvar_name, 0);
+ }
+}
+
+std::chrono::steady_clock::time_point TableRpcThrottler::throttle(LoadRelatedRpc rpc_type,
+ int64_t table_id) {
+ std::shared_lock lock(_mutex);
+ auto it = _limiters.find({rpc_type, table_id});
+ if (it == _limiters.end()) {
+ return std::chrono::steady_clock::now();
+ }
+ return it->second->reserve();
+}
+
+void TableRpcThrottler::set_qps_limit(LoadRelatedRpc rpc_type, int64_t table_id, double qps_limit) {
+ if (qps_limit <= 0) {
+ return;
+ }
+
+ std::unique_lock lock(_mutex);
+ auto key = std::make_pair(rpc_type, table_id);
+ auto it = _limiters.find(key);
+ if (it != _limiters.end()) {
+ it->second->update_qps(qps_limit);
+ } else {
+ _limiters[key] = std::make_unique<StrictQpsLimiter>(qps_limit);
+ // Update bvar count
+ size_t idx = static_cast<size_t>(rpc_type);
+ if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ size_t count = 0;
+ for (const auto& [k, _] : _limiters) {
+ if (k.first == rpc_type) {
+ ++count;
+ }
+ }
+ _throttled_table_counts[idx]->set_value(count);
+ }
+ }
+
+ LOG(INFO) << "[ms-throttle] set table QPS limit: rpc=" << load_related_rpc_name(rpc_type)
+ << ", table_id=" << table_id << ", qps_limit=" << qps_limit;
+}
+
+void TableRpcThrottler::remove_qps_limit(LoadRelatedRpc rpc_type, int64_t table_id) {
+ std::unique_lock lock(_mutex);
+ auto key = std::make_pair(rpc_type, table_id);
+ auto it = _limiters.find(key);
+ if (it != _limiters.end()) {
+ _limiters.erase(it);
+ // Update bvar count
+ size_t idx = static_cast<size_t>(rpc_type);
+ if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ size_t count = 0;
+ for (const auto& [k, _] : _limiters) {
+ if (k.first == rpc_type) {
+ ++count;
+ }
+ }
+ _throttled_table_counts[idx]->set_value(count);
+ }
+
+ LOG(INFO) << "[ms-throttle] removed table QPS limit: rpc="
+ << load_related_rpc_name(rpc_type) << ", table_id=" << table_id;
+ }
+}
+
+double TableRpcThrottler::get_qps_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+ std::shared_lock lock(_mutex);
+ auto it = _limiters.find({rpc_type, table_id});
+ if (it != _limiters.end()) {
+ return it->second->get_qps();
+ }
+ return 0;
+}
+
+bool TableRpcThrottler::has_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+ std::shared_lock lock(_mutex);
+ return _limiters.find({rpc_type, table_id}) != _limiters.end();
+}
+
+size_t TableRpcThrottler::get_throttled_table_count(LoadRelatedRpc rpc_type) const {
+ size_t idx = static_cast<size_t>(rpc_type);
+ if (idx >= static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ return 0;
+ }
+ return _throttled_table_counts[idx]->get_value();
+}
+
+std::vector<TableRpcThrottler::ThrottleEntry> TableRpcThrottler::get_all_throttled_entries() const {
+ std::shared_lock lock(_mutex);
+ std::vector<ThrottleEntry> entries;
+ entries.reserve(_limiters.size());
+ for (const auto& [key, limiter] : _limiters) {
+ entries.push_back({key.first, key.second, limiter->get_qps()});
+ }
+ return entries;
+}
+
+// ============== MSBackpressureHandler ==============
+
+MSBackpressureHandler::MSBackpressureHandler(TableRpcQpsRegistry* qps_registry,
+ TableRpcThrottler* throttler)
+ : _qps_registry(qps_registry),
+ _throttler(throttler),
+ _stop_latch(1),
+ _last_ms_busy_time(std::chrono::steady_clock::time_point::min()) {
+ // Initialize state machine with config values
+ RpcThrottleParams throttle_params {
+ .top_k = config::ms_backpressure_upgrade_top_k,
+ .ratio = config::ms_backpressure_throttle_ratio,
+ .floor_qps = config::ms_rpc_table_qps_limit_floor,
+ };
+ _state_machine = std::make_unique<RpcThrottleStateMachine>(throttle_params);
+
+ // Initialize coordinator with config values
+ // Coordinator uses ticks where 1 tick = 1 millisecond (fixed unit)
+ // This allows tick_interval_ms to change at runtime without affecting correctness
+ ThrottleCoordinatorParams coordinator_params {
+ .upgrade_cooldown_ticks = config::ms_backpressure_upgrade_interval_ms,
+ .downgrade_after_ticks = config::ms_backpressure_downgrade_interval_ms,
+ };
+ _coordinator = std::make_unique<RpcThrottleCoordinator>(coordinator_params);
+
+ auto st = Thread::create(
+ "MSBackpressureHandler", "tick_thread", [this]() { this->_tick_thread_callback(); },
+ &_tick_thread);
+ if (!st.ok()) {
+ LOG(WARNING) << "[ms-throttle] failed to create tick thread: " << st;
+ } else {
+ LOG(INFO) << "[ms-throttle] handler started: upgrade_cooldown="
+ << config::ms_backpressure_upgrade_interval_ms
+ << "ms, downgrade_interval=" << config::ms_backpressure_downgrade_interval_ms
+ << "ms";
+ }
+}
+
+MSBackpressureHandler::~MSBackpressureHandler() {
+ _stop_latch.count_down();
+ if (_tick_thread) {
+ _tick_thread->join();
+ }
+}
+
+void MSBackpressureHandler::_tick_thread_callback() {
+ // Fixed tick interval: 1 second. Since 1 tick = 1 ms, advance by 1000 ticks each iteration.
+ constexpr int kTickIntervalMs = 1000;
+ while (!_stop_latch.wait_for(std::chrono::milliseconds(kTickIntervalMs))) {
+ _advance_time(kTickIntervalMs);
+ }
+}
+
+void MSBackpressureHandler::_advance_time(int ticks) {
+ if (!config::enable_ms_backpressure_handling) {
+ return;
+ }
+
+ // Advance coordinator time; if downgrade is triggered, handle it
+ if (_coordinator->tick(ticks)) {
+ LOG(INFO) << "[ms-throttle] triggering downgrade, upgrade_level="
+ << _state_machine->upgrade_level();
+
+ auto actions = _state_machine->on_downgrade();
+ _apply_actions(actions);
+ _coordinator->set_has_pending_upgrades(_state_machine->upgrade_level() > 0);
+
+ g_backpressure_downgrade_count << 1;
+ }
+}
+
+bool MSBackpressureHandler::on_ms_busy() {
Review Comment:
**Concurrency concern (Medium): TOCTOU race between `on_ms_busy()` and `_advance_time()`**
Both `on_ms_busy()` (called from RPC threads) and `_advance_time()` (called from the tick thread) perform a multi-step sequence: coordinator decision → state machine mutation → `_apply_actions()` → coordinator feedback (`set_has_pending_upgrades`). Each individual call to `_coordinator` and `_state_machine` is internally synchronized, but the **sequence is not atomic**.
Possible interleaving:
1. Thread A (`on_ms_busy`): `_coordinator->report_ms_busy()` → returns `true` (upgrade)
2. Thread B (`_advance_time`): `_coordinator->tick()` → returns `true` (downgrade)
3. Thread B: `_state_machine->on_downgrade()` — may undo a previous upgrade
4. Thread A: `_state_machine->on_upgrade(snapshot)` — applies upgrade
5. Both set `_coordinator->set_has_pending_upgrades()` with potentially stale state
The system is self-correcting (next `MS_BUSY` re-triggers), so this is **not a crash bug**, but can cause momentary incorrect throttle limits. Consider wrapping the entire upgrade/downgrade sequence in a handler-level mutex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4044788647
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153365724
# Cloud UT Coverage Report
Increment line coverage `9.68% (3/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/e11f6697a68de51291f93879176dc7822361e9ed_e11f6697a68de51291f93879176dc7822361e9ed_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/e11f6697a68de51291f93879176dc7822361e9ed_e11f6697a68de51291f93879176dc7822361e9ed_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.64% (1797/2285) |
| Line Coverage | 64.42% (32335/50197) |
| Region Coverage | 65.27% (16203/24825) |
| Branch Coverage | 55.68% (8630/15498) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3885095915
# BE UT Coverage Report
Increment line coverage `61.31% (618/1008)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.83% (19544/36995) |
| Line Coverage | 36.31% (182073/501480) |
| Region Coverage | 32.72% (141422/432179) |
| Branch Coverage | 33.70% (61194/181565) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4152910540
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4167635039
<details>
<summary>TPC-H: <b>Total hot run time: 35362 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3e89cf3a7ca718c2d8a504573d9e974edf607060, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17648 4540 4325 4325
q2 q3 10856 985 683 683
q4 4809 501 345 345
q5 7791 880 550 550
q6 238 225 175 175
q7 924 871 625 625
q8 9511 1000 771 771
q9 9051 6544 7610 6544
q10 9016 4135 3728 3728
q11 594 343 357 343
q12 835 679 564 564
q13 18099 4931 4013 4013
q14 498 477 446 446
q15 q16 581 551 461 461
q17 977 963 761 761
q18 6618 6054 5783 5783
q19 1259 1335 886 886
q20 1538 1411 1465 1411
q21 4876 3008 2576 2576
q22 476 396 372 372
Total cold run time: 106195 ms
Total hot run time: 35362 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5391 5352 5262 5262
q2 q3 1978 2404 2049 2049
q4 1037 1408 907 907
q5 4584 4897 4769 4769
q6 231 210 157 157
q7 8050 7885 7736 7736
q8 3264 3672 3338 3338
q9 28047 27844 27710 27710
q10 4772 5114 4571 4571
q11 1100 1028 991 991
q12 623 705 538 538
q13 4646 5248 4254 4254
q14 507 503 476 476
q15 q16 579 594 561 561
q17 3948 4046 3910 3910
q18 7580 7308 7045 7045
q19 1233 1259 1232 1232
q20 2361 2439 2282 2282
q21 14066 13755 13613 13613
q22 596 550 518 518
Total cold run time: 94593 ms
Total hot run time: 91919 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3884145907
# FE UT Coverage Report
Increment line coverage ` 100.00% (10/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4171591724
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4169002824
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.57% (27283/37086) |
| Line Coverage | 57.15% (293568/513696) |
| Region Coverage | 54.24% (243841/449588) |
| Branch Coverage | 56.04% (105923/189017) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4047159865
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153057622
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168431577
run cloudut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168434876
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4411713072
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #60344:
URL: https://github.com/apache/doris/pull/60344#discussion_r3008659593
##########
cloud/test/rate_limiter_test.cpp:
##########
@@ -228,7 +228,7 @@ TEST(RateLimiterTest, TestAdjustLimitInfluence1) {
ASSERT_TRUE(meta_service->rate_limiter()->set_rate_limit(0));
std::this_thread::sleep_for(std::chrono::seconds(1));
mock_parallel_rpc(mock_get_cluster, meta_service.get(), mock_cloud_unique_id_0,
- MetaServiceCode::MAX_QPS_LIMIT, 1);
+ MetaServiceCode::MS_TOO_BUSY, 1);
Review Comment:
MS_TOO_BUSY and MAX_QPS_LIMIT are not equivalent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153931882
<details>
<summary>TPC-DS: <b>Total hot run time: 169708 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e11f6697a68de51291f93879176dc7822361e9ed, data reload: false
query5 4352 652 520 520
query6 347 229 234 229
query7 4225 470 263 263
query8 345 240 230 230
query9 8735 2762 2738 2738
query10 535 365 334 334
query11 7016 5120 4860 4860
query12 180 133 126 126
query13 1279 477 353 353
query14 5790 3679 3488 3488
query14_1 2966 2877 2834 2834
query15 204 201 179 179
query16 991 461 459 459
query17 906 730 620 620
query18 2459 465 362 362
query19 214 227 183 183
query20 127 127 124 124
query21 218 136 111 111
query22 13302 14098 14231 14098
query23 17162 16565 16129 16129
query23_1 16433 16142 15890 15890
query24 7176 1631 1220 1220
query24_1 1239 1253 1251 1251
query25 617 487 410 410
query26 1240 273 154 154
query27 2764 485 299 299
query28 4523 1850 1855 1850
query29 809 562 478 478
query30 289 226 195 195
query31 1013 958 888 888
query32 88 72 71 71
query33 536 342 283 283
query34 893 868 505 505
query35 637 687 606 606
query36 1085 1126 993 993
query37 141 97 82 82
query38 2959 2943 2898 2898
query39 850 834 808 808
query39_1 792 797 807 797
query40 234 157 137 137
query41 63 64 61 61
query42 260 256 255 255
query43 243 255 227 227
query44
query45 199 190 180 180
query46 942 989 603 603
query47 2135 2184 2073 2073
query48 321 321 230 230
query49 639 482 396 396
query50 711 303 212 212
query51 4090 4098 4057 4057
query52 263 270 257 257
query53 289 337 289 289
query54 305 284 279 279
query55 94 88 82 82
query56 300 319 311 311
query57 1915 1743 1764 1743
query58 285 272 271 271
query59 2823 2938 2737 2737
query60 346 331 329 329
query61 161 150 155 150
query62 627 589 553 553
query63 314 286 271 271
query64 5010 1293 1024 1024
query65
query66 1455 453 359 359
query67 24289 24526 24274 24274
query68
query69 409 318 287 287
query70 933 941 922 922
query71 347 305 306 305
query72 2851 2474 2486 2474
query73 552 558 328 328
query74 9671 9614 9420 9420
query75 2887 2768 2473 2473
query76 2283 1044 663 663
query77 369 378 312 312
query78 10961 11132 10516 10516
query79 1700 763 564 564
query80 1360 636 541 541
query81 561 257 223 223
query82 987 157 121 121
query83 340 275 238 238
query84 307 115 109 109
query85 951 490 449 449
query86 402 309 307 307
query87 3202 3120 3019 3019
query88 3589 2669 2661 2661
query89 417 383 346 346
query90 2043 187 176 176
query91 170 171 139 139
query92 81 78 72 72
query93 1029 879 501 501
query94 655 328 298 298
query95 612 406 332 332
query96 656 524 231 231
query97 2441 2455 2380 2380
query98 235 220 217 217
query99 1008 990 911 911
Total cold run time: 252156 ms
Total hot run time: 169708 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3883975731
# Cloud UT Coverage Report
Increment line coverage `9.68% (3/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.29% (1796/2265) |
| Line Coverage | 64.86% (32023/49369) |
| Region Coverage | 65.56% (15992/24394) |
| Branch Coverage | 56.07% (8505/15168) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4047642968
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3c9a97022829d3651babcc1963eeafe0f5bfb675_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3c9a97022829d3651babcc1963eeafe0f5bfb675_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.36% (26936/36718) |
| Line Coverage | 56.66% (287410/507252) |
| Region Coverage | 53.86% (239237/444188) |
| Branch Coverage | 55.59% (103323/185867) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3883842469
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168416344
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.57% (27286/37086) |
| Line Coverage | 57.14% (293528/513696) |
| Region Coverage | 54.21% (243731/449588) |
| Branch Coverage | 56.02% (105890/189017) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4171592095
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168158237
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.99% (20064/37867) |
| Line Coverage | 36.54% (188291/515240) |
| Region Coverage | 32.82% (146214/445472) |
| Branch Coverage | 33.94% (63963/188451) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153871462
<details>
<summary>TPC-H: <b>Total hot run time: 26863 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e11f6697a68de51291f93879176dc7822361e9ed, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17670 4559 4435 4435
q2 q3 10729 805 529 529
q4 4735 363 251 251
q5 8111 1227 997 997
q6 224 174 144 144
q7 794 860 683 683
q8 10556 1515 1361 1361
q9 6542 4786 4780 4780
q10 6294 1934 1677 1677
q11 499 255 239 239
q12 728 591 459 459
q13 18068 2822 1960 1960
q14 238 239 209 209
q15 q16 730 748 669 669
q17 736 852 460 460
q18 6221 5397 5185 5185
q19 1117 987 621 621
q20 553 479 377 377
q21 4350 2133 1535 1535
q22 370 345 292 292
Total cold run time: 99265 ms
Total hot run time: 26863 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4745 4663 4577 4577
q2 q3 3864 4349 3859 3859
q4 898 1211 811 811
q5 4047 4457 4351 4351
q6 180 171 142 142
q7 1803 1636 1534 1534
q8 2528 2751 2591 2591
q9 7843 7636 7461 7461
q10 3823 3978 3613 3613
q11 508 442 442 442
q12 592 618 446 446
q13 2465 3160 2124 2124
q14 289 312 287 287
q15 q16 758 822 718 718
q17 1231 1407 1376 1376
q18 7279 6960 6735 6735
q19 908 955 933 933
q20 2105 2146 2020 2020
q21 3970 3465 3360 3360
q22 464 429 386 386
Total cold run time: 50300 ms
Total hot run time: 47766 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60344:
URL: https://github.com/apache/doris/pull/60344#discussion_r3008312337
##########
be/src/cloud/cloud_throttle_state_machine.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_throttle_state_machine.h"
+
+#include <glog/logging.h>
+
+#include <algorithm>
+
+namespace doris::cloud {
+
+// Display names for LoadRelatedRpc types
+static constexpr std::string_view LOAD_RELATED_RPC_NAMES[] = {
+ "prepare_rowset", "commit_rowset", "update_tmp_rowset", "update_packed_file_info",
+ "update_delete_bitmap"};
+
+std::string_view load_related_rpc_name(LoadRelatedRpc rpc) {
+ size_t idx = static_cast<size_t>(rpc);
+ if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ return LOAD_RELATED_RPC_NAMES[idx];
+ }
+ return "unknown";
+}
+
+// ============== RpcThrottleStateMachine ==============
+
+RpcThrottleStateMachine::RpcThrottleStateMachine(RpcThrottleParams params) : _params(params) {
+ LOG(INFO) << "[ms-throttle] state machine initialized: top_k=" << params.top_k
+ << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+void RpcThrottleStateMachine::update_params(RpcThrottleParams params) {
+ std::lock_guard lock(_mtx);
+ _params = params;
+ LOG(INFO) << "[ms-throttle] state machine params updated: top_k=" << params.top_k
+ << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_upgrade(
+ const std::vector<RpcQpsSnapshot>& qps_snapshot) {
+ std::lock_guard lock(_mtx);
+
+ UpgradeRecord record;
+ std::vector<RpcThrottleAction> actions;
+
+ double ratio = _params.ratio;
+ double floor_qps = _params.floor_qps;
+
+ // Caller is responsible for providing top-k snapshot per RPC type.
+ // State machine simply applies throttling to every entry in the snapshot.
+ for (const auto& snapshot : qps_snapshot) {
+ auto key = std::make_pair(snapshot.rpc_type, snapshot.table_id);
+
+ double old_limit = 0.0;
+ auto limit_it = _current_limits.find(key);
+ if (limit_it != _current_limits.end()) {
+ old_limit = limit_it->second;
+ }
+
+ double new_limit;
+ if (old_limit > 0) {
+ // Already has a limit, reduce it further
+ new_limit = old_limit * ratio;
+ } else {
+ // No limit yet, set based on current QPS
+ new_limit = snapshot.current_qps * ratio;
+ }
+
+ // Apply floor
+ new_limit = std::max(new_limit, floor_qps);
+
+ // Only apply if it's actually limiting
+ if (new_limit < snapshot.current_qps || old_limit > 0) {
+ RpcThrottleAction action {
+ .type = RpcThrottleAction::Type::SET_LIMIT,
+ .rpc_type = snapshot.rpc_type,
+ .table_id = snapshot.table_id,
+ .qps_limit = new_limit,
+ };
+ actions.push_back(action);
+ record.changes[key] = {old_limit, new_limit};
+ _current_limits[key] = new_limit;
+
+ LOG(INFO) << "[ms-throttle] upgrade: rpc=" << load_related_rpc_name(snapshot.rpc_type)
+ << ", table_id=" << snapshot.table_id
+ << ", current_qps=" << snapshot.current_qps << ", old_limit=" << old_limit
+ << ", new_limit=" << new_limit;
+ }
+ }
+
+ if (!record.changes.empty()) {
+ _upgrade_history.push_back(std::move(record));
+ }
+
+ LOG(INFO) << "[ms-throttle] on_upgrade done: actions=" << actions.size()
+ << ", upgrade_level=" << _upgrade_history.size()
+ << ", snapshot_size=" << qps_snapshot.size();
+
+ return actions;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_downgrade() {
+ std::lock_guard lock(_mtx);
+
+ std::vector<RpcThrottleAction> actions;
+
+ if (_upgrade_history.empty()) {
+ LOG(INFO) << "[ms-throttle] on_downgrade skipped: no upgrade history";
+ return actions;
+ }
+
+ // Undo the most recent upgrade
+ const auto& record = _upgrade_history.back();
+
+ for (const auto& [key, limits] : record.changes) {
+ const auto& [rpc_type, table_id] = key;
+ double old_limit = limits.first;
+
+ if (old_limit > 0) {
+ // Restore the previous limit
+ RpcThrottleAction action {
+ .type = RpcThrottleAction::Type::SET_LIMIT,
+ .rpc_type = rpc_type,
+ .table_id = table_id,
+ .qps_limit = old_limit,
+ };
+
+ actions.push_back(action);
+ _current_limits[key] = old_limit;
+
+ LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+ << ", table_id=" << table_id << ", restored_limit=" << old_limit;
+ } else {
+ // No previous limit, remove it entirely
+ RpcThrottleAction action {
+ .type = RpcThrottleAction::Type::REMOVE_LIMIT,
+ .rpc_type = rpc_type,
+ .table_id = table_id,
+ };
+
+ actions.push_back(action);
+ _current_limits.erase(key);
+
+ LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+ << ", table_id=" << table_id << ", removed limit";
+ }
+ }
+
+ _upgrade_history.pop_back();
+
+ LOG(INFO) << "[ms-throttle] on_downgrade done: actions=" << actions.size()
+ << ", upgrade_level=" << _upgrade_history.size();
+
+ return actions;
+}
+
+size_t RpcThrottleStateMachine::upgrade_level() const {
+ std::lock_guard lock(_mtx);
+ return _upgrade_history.size();
+}
+
+double RpcThrottleStateMachine::get_current_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+ std::lock_guard lock(_mtx);
+ auto it = _current_limits.find({rpc_type, table_id});
+ if (it != _current_limits.end()) {
+ return it->second;
+ }
+ return 0.0;
+}
+
+RpcThrottleParams RpcThrottleStateMachine::get_params() const {
+ std::lock_guard lock(_mtx);
+ return _params;
+}
+
+// ============== RpcThrottleCoordinator ==============
+
+RpcThrottleCoordinator::RpcThrottleCoordinator(ThrottleCoordinatorParams params) : _params(params) {
+ LOG(INFO) << "[ms-throttle] coordinator initialized: upgrade_cooldown_ticks="
+ << params.upgrade_cooldown_ticks
+ << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+void RpcThrottleCoordinator::update_params(ThrottleCoordinatorParams params) {
+ std::lock_guard lock(_mtx);
+ _params = params;
+ LOG(INFO) << "[ms-throttle] coordinator params updated: upgrade_cooldown_ticks="
+ << params.upgrade_cooldown_ticks
+ << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+bool RpcThrottleCoordinator::report_ms_busy() {
+ std::lock_guard lock(_mtx);
+
+ // Reset tick counter since last MS_BUSY
+ _ticks_since_last_ms_busy = 0;
+
+ // Check if cooldown has passed
+ if (_ticks_since_last_upgrade == -1 ||
+ _ticks_since_last_upgrade >= _params.upgrade_cooldown_ticks) {
+ // Reset upgrade counter
+ _ticks_since_last_upgrade = 0;
+ _has_pending_upgrades = true;
+
+ LOG(INFO) << "[ms-throttle] upgrade triggered: ticks_since_last_upgrade="
+ << _ticks_since_last_upgrade << ", cooldown=" << _params.upgrade_cooldown_ticks;
+ return true; // Should trigger upgrade
+ }
+ return false; // Cooling down
+}
+
+bool RpcThrottleCoordinator::tick(int ticks) {
+ std::lock_guard lock(_mtx);
+
+ // Increment tick counters
+ if (_ticks_since_last_ms_busy >= 0) {
+ _ticks_since_last_ms_busy += ticks;
+ }
+ if (_ticks_since_last_upgrade >= 0) {
+ _ticks_since_last_upgrade += ticks;
+ }
+
+ // Check if downgrade should be triggered
+ if (_has_pending_upgrades && _ticks_since_last_ms_busy >= _params.downgrade_after_ticks) {
+ // Reset for next downgrade cycle
+ _ticks_since_last_ms_busy = 0;
+
+ LOG(INFO) << "[ms-throttle] downgrade triggered: ticks_since_last_ms_busy="
Review Comment:
**Bug: Log message always prints `downgrade_after_ticks` instead of actual elapsed ticks.**
At this point, `_ticks_since_last_ms_busy` has already been reset to `0` on line 240. So the expression `_ticks_since_last_ms_busy + _params.downgrade_after_ticks` always evaluates to `0 + downgrade_after_ticks = downgrade_after_ticks`, not the actual ticks that elapsed.
Fix: capture the value before resetting:
```cpp
auto actual_ticks = _ticks_since_last_ms_busy;
_ticks_since_last_ms_busy = 0;
LOG(INFO) << "[ms-throttle] downgrade triggered: ticks_since_last_ms_busy="
<< actual_ticks << ", threshold=" << _params.downgrade_after_ticks;
```
##########
be/src/cloud/cloud_throttle_state_machine.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "cloud/cloud_throttle_state_machine.h"
+
+#include <glog/logging.h>
+
+#include <algorithm>
+
+namespace doris::cloud {
+
+// Display names for LoadRelatedRpc types
+static constexpr std::string_view LOAD_RELATED_RPC_NAMES[] = {
+ "prepare_rowset", "commit_rowset", "update_tmp_rowset", "update_packed_file_info",
+ "update_delete_bitmap"};
+
+std::string_view load_related_rpc_name(LoadRelatedRpc rpc) {
+ size_t idx = static_cast<size_t>(rpc);
+ if (idx < static_cast<size_t>(LoadRelatedRpc::COUNT)) {
+ return LOAD_RELATED_RPC_NAMES[idx];
+ }
+ return "unknown";
+}
+
+// ============== RpcThrottleStateMachine ==============
+
+RpcThrottleStateMachine::RpcThrottleStateMachine(RpcThrottleParams params) : _params(params) {
+ LOG(INFO) << "[ms-throttle] state machine initialized: top_k=" << params.top_k
+ << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+void RpcThrottleStateMachine::update_params(RpcThrottleParams params) {
+ std::lock_guard lock(_mtx);
+ _params = params;
+ LOG(INFO) << "[ms-throttle] state machine params updated: top_k=" << params.top_k
+ << ", ratio=" << params.ratio << ", floor_qps=" << params.floor_qps;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_upgrade(
+ const std::vector<RpcQpsSnapshot>& qps_snapshot) {
+ std::lock_guard lock(_mtx);
+
+ UpgradeRecord record;
+ std::vector<RpcThrottleAction> actions;
+
+ double ratio = _params.ratio;
+ double floor_qps = _params.floor_qps;
+
+ // Caller is responsible for providing top-k snapshot per RPC type.
+ // State machine simply applies throttling to every entry in the snapshot.
+ for (const auto& snapshot : qps_snapshot) {
+ auto key = std::make_pair(snapshot.rpc_type, snapshot.table_id);
+
+ double old_limit = 0.0;
+ auto limit_it = _current_limits.find(key);
+ if (limit_it != _current_limits.end()) {
+ old_limit = limit_it->second;
+ }
+
+ double new_limit;
+ if (old_limit > 0) {
+ // Already has a limit, reduce it further
+ new_limit = old_limit * ratio;
+ } else {
+ // No limit yet, set based on current QPS
+ new_limit = snapshot.current_qps * ratio;
+ }
+
+ // Apply floor
+ new_limit = std::max(new_limit, floor_qps);
+
+ // Only apply if it's actually limiting
+ if (new_limit < snapshot.current_qps || old_limit > 0) {
+ RpcThrottleAction action {
+ .type = RpcThrottleAction::Type::SET_LIMIT,
+ .rpc_type = snapshot.rpc_type,
+ .table_id = snapshot.table_id,
+ .qps_limit = new_limit,
+ };
+ actions.push_back(action);
+ record.changes[key] = {old_limit, new_limit};
+ _current_limits[key] = new_limit;
+
+ LOG(INFO) << "[ms-throttle] upgrade: rpc=" << load_related_rpc_name(snapshot.rpc_type)
+ << ", table_id=" << snapshot.table_id
+ << ", current_qps=" << snapshot.current_qps << ", old_limit=" << old_limit
+ << ", new_limit=" << new_limit;
+ }
+ }
+
+ if (!record.changes.empty()) {
+ _upgrade_history.push_back(std::move(record));
+ }
+
+ LOG(INFO) << "[ms-throttle] on_upgrade done: actions=" << actions.size()
+ << ", upgrade_level=" << _upgrade_history.size()
+ << ", snapshot_size=" << qps_snapshot.size();
+
+ return actions;
+}
+
+std::vector<RpcThrottleAction> RpcThrottleStateMachine::on_downgrade() {
+ std::lock_guard lock(_mtx);
+
+ std::vector<RpcThrottleAction> actions;
+
+ if (_upgrade_history.empty()) {
+ LOG(INFO) << "[ms-throttle] on_downgrade skipped: no upgrade history";
+ return actions;
+ }
+
+ // Undo the most recent upgrade
+ const auto& record = _upgrade_history.back();
+
+ for (const auto& [key, limits] : record.changes) {
+ const auto& [rpc_type, table_id] = key;
+ double old_limit = limits.first;
+
+ if (old_limit > 0) {
+ // Restore the previous limit
+ RpcThrottleAction action {
+ .type = RpcThrottleAction::Type::SET_LIMIT,
+ .rpc_type = rpc_type,
+ .table_id = table_id,
+ .qps_limit = old_limit,
+ };
+
+ actions.push_back(action);
+ _current_limits[key] = old_limit;
+
+ LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+ << ", table_id=" << table_id << ", restored_limit=" << old_limit;
+ } else {
+ // No previous limit, remove it entirely
+ RpcThrottleAction action {
+ .type = RpcThrottleAction::Type::REMOVE_LIMIT,
+ .rpc_type = rpc_type,
+ .table_id = table_id,
+ };
+
+ actions.push_back(action);
+ _current_limits.erase(key);
+
+ LOG(INFO) << "[ms-throttle] downgrade: rpc=" << load_related_rpc_name(rpc_type)
+ << ", table_id=" << table_id << ", removed limit";
+ }
+ }
+
+ _upgrade_history.pop_back();
+
+ LOG(INFO) << "[ms-throttle] on_downgrade done: actions=" << actions.size()
+ << ", upgrade_level=" << _upgrade_history.size();
+
+ return actions;
+}
+
+size_t RpcThrottleStateMachine::upgrade_level() const {
+ std::lock_guard lock(_mtx);
+ return _upgrade_history.size();
+}
+
+double RpcThrottleStateMachine::get_current_limit(LoadRelatedRpc rpc_type, int64_t table_id) const {
+ std::lock_guard lock(_mtx);
+ auto it = _current_limits.find({rpc_type, table_id});
+ if (it != _current_limits.end()) {
+ return it->second;
+ }
+ return 0.0;
+}
+
+RpcThrottleParams RpcThrottleStateMachine::get_params() const {
+ std::lock_guard lock(_mtx);
+ return _params;
+}
+
+// ============== RpcThrottleCoordinator ==============
+
+RpcThrottleCoordinator::RpcThrottleCoordinator(ThrottleCoordinatorParams params) : _params(params) {
+ LOG(INFO) << "[ms-throttle] coordinator initialized: upgrade_cooldown_ticks="
+ << params.upgrade_cooldown_ticks
+ << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+void RpcThrottleCoordinator::update_params(ThrottleCoordinatorParams params) {
+ std::lock_guard lock(_mtx);
+ _params = params;
+ LOG(INFO) << "[ms-throttle] coordinator params updated: upgrade_cooldown_ticks="
+ << params.upgrade_cooldown_ticks
+ << ", downgrade_after_ticks=" << params.downgrade_after_ticks;
+}
+
+bool RpcThrottleCoordinator::report_ms_busy() {
+ std::lock_guard lock(_mtx);
+
+ // Reset tick counter since last MS_BUSY
+ _ticks_since_last_ms_busy = 0;
+
+ // Check if cooldown has passed
+ if (_ticks_since_last_upgrade == -1 ||
+ _ticks_since_last_upgrade >= _params.upgrade_cooldown_ticks) {
+ // Reset upgrade counter
+ _ticks_since_last_upgrade = 0;
+ _has_pending_upgrades = true;
+
+ LOG(INFO) << "[ms-throttle] upgrade triggered: ticks_since_last_upgrade="
Review Comment:
**Bug: Log message always prints 0 for `ticks_since_last_upgrade`.**
`_ticks_since_last_upgrade` was just set to `0` on line 216, so this log always prints `0` instead of the actual number of ticks since the last upgrade (which would show how long the cooldown actually lasted).
Fix: capture before resetting:
```cpp
auto actual_ticks = _ticks_since_last_upgrade;
_ticks_since_last_upgrade = 0;
_has_pending_upgrades = true;
LOG(INFO) << "[ms-throttle] upgrade triggered: ticks_since_last_upgrade="
<< actual_ticks << ", cooldown=" << _params.upgrade_cooldown_ticks;
```
##########
be/src/cloud/config.cpp:
##########
@@ -173,5 +173,49 @@ DEFINE_mInt64(cache_read_from_peer_expired_seconds, "-1");
DEFINE_mBool(enable_file_cache_write_base_compaction_index_only, "false");
DEFINE_mBool(enable_file_cache_write_cumu_compaction_index_only, "false");
+// MS RPC rate limiting config
+DEFINE_mBool(enable_ms_rpc_host_level_rate_limit, "true");
+
+// Per-RPC QPS limit configs (per CPU core)
+// QPS limit = config_value * num_cores
+// Set to 0 to disable rate limiting for a specific RPC
+// Set to -1 to use ms_rpc_qps_default config value
+DEFINE_mInt32(ms_rpc_qps_default, "100");
+DEFINE_mInt32(ms_rpc_qps_get_tablet_meta, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_prepare_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_commit_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_update_tmp_rowset, "-1");
+DEFINE_mInt32(ms_rpc_qps_commit_txn, "-1");
+DEFINE_mInt32(ms_rpc_qps_abort_txn, "-1");
+DEFINE_mInt32(ms_rpc_qps_precommit_txn, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_obj_store_info, "-1");
+DEFINE_mInt32(ms_rpc_qps_start_tablet_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_finish_tablet_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_delete_bitmap, "-1");
+DEFINE_mInt32(ms_rpc_qps_update_delete_bitmap, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_delete_bitmap_update_lock, "-1");
+DEFINE_mInt32(ms_rpc_qps_remove_delete_bitmap_update_lock, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_instance, "-1");
+DEFINE_mInt32(ms_rpc_qps_prepare_restore_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_commit_restore_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_finish_restore_job, "-1");
+DEFINE_mInt32(ms_rpc_qps_list_snapshots, "-1");
+DEFINE_mInt32(ms_rpc_qps_get_cluster_status, "-1");
+DEFINE_mInt32(ms_rpc_qps_update_packed_file_info, "-1");
+
+// Table-level backpressure handling config
+DEFINE_mBool(enable_ms_backpressure_handling, "true");
Review Comment:
**Doc mismatch**: The PR description table says `enable_ms_backpressure_handling` defaults to `false`, but here it defaults to `"true"`. Please align the PR description with the code, or change the default if `false` was intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4167217808
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4046262629
# BE UT Coverage Report
Increment line coverage `61.17% (619/1012)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3c9a97022829d3651babcc1963eeafe0f5bfb675_3c9a97022829d3651babcc1963eeafe0f5bfb675/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3c9a97022829d3651babcc1963eeafe0f5bfb675_3c9a97022829d3651babcc1963eeafe0f5bfb675/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.68% (19749/37486) |
| Line Coverage | 36.23% (184114/508156) |
| Region Coverage | 32.38% (142321/439594) |
| Branch Coverage | 33.51% (62026/185103) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "wyxxxcat (via GitHub)" <gi...@apache.org>.
wyxxxcat commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4384935506
run check_coverage_fe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3884403194
<details>
<summary>TPC-H: <b>Total hot run time: 30385 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa, data reload: false
------ Round 1 ----------------------------------
q1 17679 4467 4272 4272
q2 2039 390 244 244
q3 10109 1308 749 749
q4 10207 793 319 319
q5 7551 2209 1958 1958
q6 205 184 150 150
q7 882 748 619 619
q8 9263 1420 1155 1155
q9 4769 4700 4617 4617
q10 6827 1937 1546 1546
q11 494 265 237 237
q12 345 383 229 229
q13 17782 4100 3220 3220
q14 231 233 223 223
q15 906 821 818 818
q16 701 679 604 604
q17 713 816 547 547
q18 7109 5985 5815 5815
q19 1106 1011 627 627
q20 513 503 393 393
q21 2575 1838 1794 1794
q22 327 289 249 249
Total cold run time: 102333 ms
Total hot run time: 30385 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4383 4324 4319 4319
q2 280 336 259 259
q3 2125 2720 2212 2212
q4 1368 1746 1307 1307
q5 4336 4218 4183 4183
q6 220 193 137 137
q7 1879 1830 1702 1702
q8 2673 2631 2489 2489
q9 7645 7429 7470 7429
q10 2810 3278 2678 2678
q11 501 439 433 433
q12 682 728 644 644
q13 3895 4764 3545 3545
q14 308 348 302 302
q15 849 810 807 807
q16 698 740 678 678
q17 1146 1339 1393 1339
q18 8283 7969 8028 7969
q19 887 884 859 859
q20 2104 2189 2065 2065
q21 4771 4677 4393 4393
q22 527 469 485 469
Total cold run time: 52370 ms
Total hot run time: 50218 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-3884489766
<details>
<summary>ClickBench: <b>Total hot run time: 28.3 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bdd55d6cb3a5862cc5ec61e49dca8fe680e0e3aa, data reload: false
query1 0.05 0.04 0.04
query2 0.10 0.05 0.05
query3 0.25 0.08 0.08
query4 1.61 0.12 0.11
query5 0.27 0.26 0.25
query6 1.16 0.68 0.66
query7 0.03 0.03 0.03
query8 0.06 0.04 0.04
query9 0.56 0.52 0.49
query10 0.55 0.54 0.54
query11 0.14 0.10 0.10
query12 0.14 0.10 0.10
query13 0.64 0.61 0.62
query14 1.06 1.06 1.05
query15 0.88 0.86 0.87
query16 0.41 0.39 0.40
query17 1.07 1.05 1.14
query18 0.22 0.22 0.21
query19 2.10 2.02 2.01
query20 0.02 0.02 0.01
query21 15.39 0.26 0.15
query22 5.43 0.06 0.05
query23 16.17 0.28 0.11
query24 1.49 0.34 0.23
query25 0.09 0.07 0.08
query26 0.15 0.14 0.15
query27 0.11 0.09 0.05
query28 3.59 1.17 0.97
query29 12.56 3.95 3.19
query30 0.27 0.14 0.12
query31 2.81 0.65 0.41
query32 3.25 0.58 0.50
query33 3.24 3.25 3.27
query34 16.04 5.41 4.73
query35 4.80 4.80 4.80
query36 0.65 0.50 0.48
query37 0.12 0.07 0.07
query38 0.08 0.04 0.05
query39 0.05 0.03 0.03
query40 0.22 0.16 0.14
query41 0.09 0.03 0.03
query42 0.05 0.03 0.03
query43 0.05 0.04 0.03
Total cold run time: 98.02 s
Total hot run time: 28.3 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4393695651
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4168636818
# Cloud UT Coverage Report
Increment line coverage `9.68% (3/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3e89cf3a7ca718c2d8a504573d9e974edf607060_3e89cf3a7ca718c2d8a504573d9e974edf607060_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.46% (1799/2293) |
| Line Coverage | 64.18% (32322/50365) |
| Region Coverage | 65.01% (16215/24942) |
| Branch Coverage | 55.46% (8640/15580) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4169096965
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4169608003
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60344_3e89cf3a7ca718c2d8a504573d9e974edf607060_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.57% (27285/37086) |
| Line Coverage | 57.15% (293553/513696) |
| Region Coverage | 54.26% (243944/449588) |
| Branch Coverage | 56.05% (105946/189017) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #60344:
URL: https://github.com/apache/doris/pull/60344#issuecomment-4153134910
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](cloud) Add rate limit for BE to MS rpc [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #60344:
URL: https://github.com/apache/doris/pull/60344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org