You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by "stdpain (via GitHub)" <gi...@apache.org> on 2026/07/13 03:07:37 UTC

[PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

stdpain opened a new pull request, #3384:
URL: https://github.com/apache/brpc/pull/3384

   ### What problem does this PR solve?
   
   Issue Number: #3383 
   
   Problem Summary:
   
   ### What is changed and the side effects?
   
    bthread::TimerThread reclaims a task's pooled Task slot only when the task is popped from the internal min-heap at its run_time. unschedule() just marks the task removed and does not free the slot. So a
     cancelled timer keeps its slot until run_time, and the live Task count grows to ~ qps * timeout even though almost all of those tasks were unscheduled long ago. Two retention paths:
   
     1. Heap retention — a task pulled into the heap is only checked for cancellation once (via try_delete() at consume time). If unscheduled afterwards, it lingers in the heap until run_time.
     2. Bucket accumulation — the timer thread only consumes buckets when it wakes, and schedule() wakes it only for an earlier task. If every pending task is far in the future, scheduled-then-unscheduled
     tasks pile up in the buckets, unconsumed and unreclaimed, until that far deadline.
   
     This is significant for RPC deadline timers with large timeouts (e.g. 100k qps × 300s ≈ ~1.9 GB held mostly by already-cancelled timers).
   
   Changed:
   
     - Heap sweep: in TimerThread::run(), after consuming buckets, drop unscheduled tasks from the heap. Triggered by a timer-thread-local heuristic (heap ≥ brpc_timer_heap_sweep_min_size, default 4096, and
     roughly doubled since the last sweep), so the unschedule() hot path is untouched (no new shared atomics). Amortized O(1) per task.
     - Periodic wakeup: cap the sleep at brpc_timer_max_wakeup_interval_ms (default 0 = disabled, legacy behavior) so the thread periodically wakes to drain buckets / sweep the heap even when all deadlines
     are far-future. An empty heap still sleeps until woken by schedule().
     - Added pending_task_count() and allocated_task_count() for observability.
     - Added unit tests for both the heap-retention and bucket-accumulation cases (each verified to fail with the corresponding fix disabled).
   
   Side effects:
   - Performance effects:
   
   - Breaking backward compatibility: 
   
   ---
   ### Check List:
   - Please make sure your changes are compilable.
   - When providing us with a new feature, it is best to add related tests.
   - Please follow [Contributor Covenant Code of Conduct](https://github.com/apache/brpc/blob/master/CODE_OF_CONDUCT.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

Posted by "stdpain (via GitHub)" <gi...@apache.org>.
stdpain commented on code in PR #3384:
URL: https://github.com/apache/brpc/pull/3384#discussion_r3578017511


##########
src/bthread/timer_thread.h:
##########
@@ -84,7 +84,22 @@ class TimerThread {
     // Get identifier of internal pthread.
     // Returns (pthread_t)0 if start() is not called yet.
     pthread_t thread_id() const { return _thread; }
-    
+
+    // Number of tasks currently buffered in the timer thread's internal
+    // min-heap (pulled out of buckets but not run yet). Exposed mainly for
+    // tests and observability. Note that tasks unscheduled after being pulled
+    // into the heap are counted until the timer thread sweeps them away.
+    int64_t pending_task_count() const {
+        return _npending.load(butil::memory_order_relaxed);
+    }
+
+    // Total number of internal Task objects the process-wide pool has ever
+    // constructed (a high-water mark that never shrinks). Shared by all
+    // TimerThread instances. Exposed for tests/observability: if unscheduled
+    // tasks are reclaimed promptly, freed slots get reused and this stays
+    // bounded instead of growing with the number of scheduled tasks.
+    static size_t allocated_task_count();

Review Comment:
   fixed



##########
src/bthread/timer_thread.h:
##########
@@ -84,7 +84,22 @@ class TimerThread {
     // Get identifier of internal pthread.
     // Returns (pthread_t)0 if start() is not called yet.
     pthread_t thread_id() const { return _thread; }
-    
+
+    // Number of tasks currently buffered in the timer thread's internal
+    // min-heap (pulled out of buckets but not run yet). Exposed mainly for
+    // tests and observability. Note that tasks unscheduled after being pulled
+    // into the heap are counted until the timer thread sweeps them away.
+    int64_t pending_task_count() const {

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

Posted by "stdpain (via GitHub)" <gi...@apache.org>.
stdpain commented on code in PR #3384:
URL: https://github.com/apache/brpc/pull/3384#discussion_r3567868767


##########
src/bthread/timer_thread.cpp:
##########
@@ -408,6 +466,12 @@ void TimerThread::run() {
             BT_VLOG << "pull again, tasks=" << tasks.size();
             continue;
         }

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on PR #3384:
URL: https://github.com/apache/brpc/pull/3384#issuecomment-4978409256

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on code in PR #3384:
URL: https://github.com/apache/brpc/pull/3384#discussion_r3577914809


##########
src/bthread/timer_thread.h:
##########
@@ -84,7 +84,22 @@ class TimerThread {
     // Get identifier of internal pthread.
     // Returns (pthread_t)0 if start() is not called yet.
     pthread_t thread_id() const { return _thread; }
-    
+
+    // Number of tasks currently buffered in the timer thread's internal
+    // min-heap (pulled out of buckets but not run yet). Exposed mainly for
+    // tests and observability. Note that tasks unscheduled after being pulled
+    // into the heap are counted until the timer thread sweeps them away.
+    int64_t pending_task_count() const {

Review Comment:
   You don't need to add this public interface, brpc test code can assess private members (through the `-fno-access-control` compiler flag)



##########
src/bthread/timer_thread.h:
##########
@@ -84,7 +84,22 @@ class TimerThread {
     // Get identifier of internal pthread.
     // Returns (pthread_t)0 if start() is not called yet.
     pthread_t thread_id() const { return _thread; }
-    
+
+    // Number of tasks currently buffered in the timer thread's internal
+    // min-heap (pulled out of buckets but not run yet). Exposed mainly for
+    // tests and observability. Note that tasks unscheduled after being pulled
+    // into the heap are counted until the timer thread sweeps them away.
+    int64_t pending_task_count() const {
+        return _npending.load(butil::memory_order_relaxed);
+    }
+
+    // Total number of internal Task objects the process-wide pool has ever
+    // constructed (a high-water mark that never shrinks). Shared by all
+    // TimerThread instances. Exposed for tests/observability: if unscheduled
+    // tasks are reclaimed promptly, freed slots get reused and this stays
+    // bounded instead of growing with the number of scheduled tasks.
+    static size_t allocated_task_count();

Review Comment:
   Same as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

Posted by "stdpain (via GitHub)" <gi...@apache.org>.
stdpain commented on code in PR #3384:
URL: https://github.com/apache/brpc/pull/3384#discussion_r3567868116


##########
test/bthread_timer_thread_unittest.cpp:
##########
@@ -254,4 +256,113 @@ TEST(TimerThreadTest, schedule_and_unschedule_in_task) {
     keeper5.expect_first_run();
 }
 
+static void noop_routine(void*) {}
+
+// Tasks that are unscheduled after being pulled into the timer thread's
+// internal heap must not stay there (occupying a pooled Task slot) until
+// their run_time. With large timeouts that would let memory grow ~ qps *
+// timeout. The timer thread should sweep them out and keep the heap bounded.
+TEST(TimerThreadTest, sweep_unscheduled_tasks_in_heap) {
+    // Lower the sweep threshold so the test doesn't need a huge heap.
+    GFLAGS_NAMESPACE::SetCommandLineOption(
+        "brpc_timer_heap_sweep_min_size", "512");

Review Comment:
   UT update



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm merged PR #3384:
URL: https://github.com/apache/brpc/pull/3384


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [PR] Reclaim unscheduled timer tasks instead of holding slots until run_time (brpc)

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3384:
URL: https://github.com/apache/brpc/pull/3384#discussion_r3567849626


##########
test/bthread_timer_thread_unittest.cpp:
##########
@@ -254,4 +256,113 @@ TEST(TimerThreadTest, schedule_and_unschedule_in_task) {
     keeper5.expect_first_run();
 }
 
+static void noop_routine(void*) {}
+
+// Tasks that are unscheduled after being pulled into the timer thread's
+// internal heap must not stay there (occupying a pooled Task slot) until
+// their run_time. With large timeouts that would let memory grow ~ qps *
+// timeout. The timer thread should sweep them out and keep the heap bounded.
+TEST(TimerThreadTest, sweep_unscheduled_tasks_in_heap) {
+    // Lower the sweep threshold so the test doesn't need a huge heap.
+    GFLAGS_NAMESPACE::SetCommandLineOption(
+        "brpc_timer_heap_sweep_min_size", "512");
+
+    bthread::TimerThread timer_thread;
+    ASSERT_EQ(0, timer_thread.start(NULL));
+
+    // Run far enough in the future that these tasks never fire on their own.
+    const timespec far = butil::seconds_from_now(100000);
+    const size_t kBatch = 2000;
+    const size_t kRounds = 20;
+
+    int64_t max_pending = 0;
+    for (size_t r = 0; r < kRounds; ++r) {
+        std::vector<bthread::TimerThread::TaskId> ids;
+        ids.reserve(kBatch);
+        for (size_t i = 0; i < kBatch; ++i) {
+            ids.push_back(timer_thread.schedule(noop_routine, NULL, far));
+        }
+        // A near-term task forces the timer thread to wake up and consume the
+        // buckets, so the far tasks above land in the heap (alive).
+        timer_thread.schedule(noop_routine, NULL,
+                              butil::milliseconds_from_now(1));
+        usleep(20000);  // let the timer thread consume the buckets
+
+        // Now unschedule the far tasks: they become dead-in-heap, exactly the
+        // case that used to linger until run_time.
+        for (size_t i = 0; i < ids.size(); ++i) {
+            timer_thread.unschedule(ids[i]);
+        }
+        // Another near-term task wakes the timer thread again, triggering the
+        // sweep that reclaims the dead tasks.
+        timer_thread.schedule(noop_routine, NULL,
+                              butil::milliseconds_from_now(1));
+        usleep(20000);
+
+        const int64_t pending = timer_thread.pending_task_count();
+        LOG(INFO) << "round=" << r << " pending=" << pending;
+        max_pending = std::max(max_pending, pending);
+    }
+
+    // Without the sweep, pending would grow to ~ kBatch * kRounds (40000).
+    // With it, the heap is bounded by roughly a couple of sweep thresholds.
+    EXPECT_LT(max_pending, (int64_t)(kBatch * kRounds) / 4)
+        << "dead tasks are not being reclaimed from the heap";
+
+    timer_thread.stop_and_join();
+}
+
+// When every pending task is far in the future, the nearest run_time never
+// arrives, so the timer thread used to sleep for the whole duration. Newly
+// scheduled-then-unscheduled tasks would then pile up in the buckets (never
+// consumed, so never reclaimed) for that entire time. The capped wakeup makes
+// the timer thread wake up periodically to drain the buckets, keeping the pool
+// bounded regardless of how far the timeouts are.
+TEST(TimerThreadTest, periodic_wakeup_reclaims_bucket_tasks) {
+    // Wake up frequently and keep any heap slippage swept too.
+    GFLAGS_NAMESPACE::SetCommandLineOption(
+        "brpc_timer_max_wakeup_interval_ms", "50");
+    GFLAGS_NAMESPACE::SetCommandLineOption(
+        "brpc_timer_heap_sweep_min_size", "512");

Review Comment:
   This test changes global gflags (brpc_timer_max_wakeup_interval_ms and brpc_timer_heap_sweep_min_size) but never restores them, which can affect subsequent tests. Save previous values and restore them on scope exit; also check SetCommandLineOption return values so failures to set the flag don’t turn into confusing test flakes.



##########
test/bthread_timer_thread_unittest.cpp:
##########
@@ -254,4 +256,113 @@ TEST(TimerThreadTest, schedule_and_unschedule_in_task) {
     keeper5.expect_first_run();
 }
 
+static void noop_routine(void*) {}
+
+// Tasks that are unscheduled after being pulled into the timer thread's
+// internal heap must not stay there (occupying a pooled Task slot) until
+// their run_time. With large timeouts that would let memory grow ~ qps *
+// timeout. The timer thread should sweep them out and keep the heap bounded.
+TEST(TimerThreadTest, sweep_unscheduled_tasks_in_heap) {
+    // Lower the sweep threshold so the test doesn't need a huge heap.
+    GFLAGS_NAMESPACE::SetCommandLineOption(
+        "brpc_timer_heap_sweep_min_size", "512");

Review Comment:
   This test changes the global gflag brpc_timer_heap_sweep_min_size but never restores it, which can leak configuration into later tests and cause order-dependent failures. Save the previous value and restore it on scope exit; also assert that SetCommandLineOption succeeded so the test doesn’t silently run with defaults.



##########
src/bthread/timer_thread.cpp:
##########
@@ -408,6 +466,12 @@ void TimerThread::run() {
             BT_VLOG << "pull again, tasks=" << tasks.size();
             continue;
         }

Review Comment:
   pending_task_count() is updated only on the non-pull_again path. When pull_again is taken, _npending can stay stale even though the heap content may change across iterations, making the new observability API misleading (and potentially flaky for tests that sample it). Update _npending before continuing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org