You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@brpc.apache.org by "hjwsm1989 (via GitHub)" <gi...@apache.org> on 2026/04/22 16:50:09 UTC

[I] bvar: sampler busy-loops at ~1 kHz after switch to cpuwide_time_ns (regression from #3268) (brpc)

hjwsm1989 opened a new issue, #3277:
URL: https://github.com/apache/brpc/issues/3277

   ### Describe the bug
   
   Since commit 12fb539a ("Use monotonic time instead of wall time" / #3268, merged around 1.17.0-rc1), the bvar sampler thread no longer sleeps between sampling rounds and prints
   
       sampler.cpp:192 bvar is busy at sampling for 2 seconds!
   
   continuously (tens of thousands of log lines per minute on a process with a moderate number of bvars).
   
   Root cause is a unit-conversion mistake in `src/bvar/detail/sampler.cpp::SamplerCollector::run()`:
   
   ```cpp
   while (!_stop) {
       int64_t abstime = butil::cpuwide_time_ns();   // ns  (was gettimeofday_us() / us)
       ...
       int64_t now = butil::cpuwide_time_ns();       // ns
       _cumulated_time_us += now - abstime;
       abstime += 1000000L;                          // BUG: + 1 ms, not + 1 s
       while (abstime > now) {
           ::usleep(abstime - now);                  // BUG: usleep() takes us, value is ns
           slept = true;
           now = butil::cpuwide_time_ns();
       }
       if (slept) { consecutive_nosleep = 0; }
       else if (++consecutive_nosleep >= WARN_NOSLEEP_THRESHOLD) { LOG(WARNING) ... }
   }
   ```
   
   PR #3268 switched the three time-source calls from `gettimeofday_us()` (microseconds) to `cpuwide_time_ns()` (nanoseconds), but left `abstime += 1000000L` and `::usleep(abstime - now)` untouched. The intended "sample once per second" becomes "advance abstime by 1 ms"; the inner `while` loop almost never enters (so `slept` stays `false`), and after two consecutive nosleep iterations the warning fires. In practice the sampler runs at ~1 kHz instead of 1 Hz, burning CPU and flooding logs.
   
   ### To Reproduce
   
   1. Build any service linking brpc >= 1.17.0-rc1 (commit 12fb539a or newer).
   2. Register any bvar (e.g. `bvar::LatencyRecorder`).
   3. Start the process and tail its log:
   
      ```bash
      grep "bvar is busy at sampling" <logfile> | wc -l
      ```
   
      Observed: tens of thousands of warnings per minute.
   
   ### Expected behavior
   
   The sampler runs once per second; no "busy" warning unless the host is truly overloaded (the prior <=1.16.x behavior).
   
   ### Versions
   
   - brpc: 1.17.0-rc1 / current master (commit 12fb539a and descendants)
   - OS: Linux 5.14 (x86_64)
   - Compiler: gcc 12, C++20
   
   ### Fix
   
   Follow-up PR to be linked below.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org


Re: [I] bvar: sampler busy-loops at ~1 kHz after switch to cpuwide_time_ns (regression from #3268) (brpc)

Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright closed issue #3277: bvar: sampler busy-loops at ~1 kHz after switch to cpuwide_time_ns (regression from #3268)
URL: https://github.com/apache/brpc/issues/3277


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org