You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sahildevgon (via GitHub)" <gi...@apache.org> on 2026/02/16 07:14:38 UTC
[PR] [Enhancement](metrics)Added CPU usage metrics [doris]
sahildevgon opened a new pull request, #60777:
URL: https://github.com/apache/doris/pull/60777
### What problem does this PR solve?
Enhanced Apache Doris Frontend (FE) to expose comprehensive CPU and system metrics via the `/metrics` endpoint. Previously, CPU metrics were not exposed, limiting observability and troubleshooting capabilities.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [X] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [X] Other reason <!-- Enhanced existing SystemMetrics to additionally fetch cpu usage metrics -->
- Behavior changed:
- [X] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [X] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3995122508
<details>
<summary>TPC-H: <b>Total hot run time: 28747 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d6a691021c9aea4150fa2352fad9346ebcbc5e9, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17625 4481 4314 4314
q2 q3 10645 779 520 520
q4 4673 363 265 265
q5 7559 1208 1024 1024
q6 173 178 148 148
q7 773 857 654 654
q8 9301 1445 1294 1294
q9 4922 4784 4728 4728
q10 6821 1881 1624 1624
q11 463 262 249 249
q12 707 579 467 467
q13 17790 4216 3408 3408
q14 237 236 211 211
q15 965 790 787 787
q16 752 730 692 692
q17 737 893 411 411
q18 6219 5371 5275 5275
q19 1256 975 628 628
q20 510 497 391 391
q21 4677 1859 1414 1414
q22 348 287 243 243
Total cold run time: 97153 ms
Total hot run time: 28747 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4436 4351 4361 4351
q2 q3 1753 2178 1708 1708
q4 847 1156 766 766
q5 4043 4333 4339 4333
q6 179 177 139 139
q7 1739 1582 1495 1495
q8 2409 2643 2518 2518
q9 7214 7061 7099 7061
q10 2529 2729 2320 2320
q11 497 442 405 405
q12 476 565 437 437
q13 3753 4251 3408 3408
q14 266 282 260 260
q15 832 780 807 780
q16 685 755 692 692
q17 1099 1419 1259 1259
q18 6848 6672 6608 6608
q19 838 822 863 822
q20 2018 2101 1946 1946
q21 3899 3463 3310 3310
q22 467 434 370 370
Total cold run time: 46827 ms
Total hot run time: 44988 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4025322413
hi @morningman ,
Reg. point 1. [Critical] Split Unrelated Changes into Separate PRs :
- I have opened a New PR for the bug fix : https://github.com/apache/doris/pull/61145 (Under review).
Reg. points 2,3,4,6 :
- I have posted new commits to align the changes with your review comments.
Reg. point 5. [Low] Consider Using Getter Methods for Field Access.
- All 17 field accesses in MetricRepo.java use direct field access — the existing TCP fields (tcpRetransSegs, tcpInErrs,
etc.) and memory fields (memTotal, memFree, etc.) all follow the same pattern.
- Introducing getters only for the new CPU fields would create inconsistency within the same class.
- If you suggest I can post a separate PR for introducing getters for all fields.
PS: The build of this PR might fail until we have the [Fix] PR : https://github.com/apache/doris/pull/61145. (For which i am following up on separate thread)
PTAL and requesting further review.
thanks,
Sahil Devgon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4073997452
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4024782972
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4069524472
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60777:
URL: https://github.com/apache/doris/pull/60777#discussion_r2900482620
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -145,4 +204,139 @@ private void updateMemoryMetrics() {
}
}
+ private void updateCpuMetrics() {
+ String procFile = "/proc/stat";
+ if (FeConstants.runningUnitTest) {
+ procFile = getClass().getClassLoader().getResource("data/stat_normal").getFile();
+ }
+
+ try (FileReader fileReader = new FileReader(procFile);
+ BufferedReader br = new BufferedReader(fileReader)) {
+ String line;
+ boolean cpuLineFound = false;
+
+ // Store previous values for delta calculation
+ long prevTotal = prevCpuUser + prevCpuNice + prevCpuSystem + prevCpuIdle
+ + prevCpuIowait + prevCpuIrq + prevCpuSoftirq + prevCpuSteal;
+
+ while ((line = br.readLine()) != null) {
+ if (line.startsWith("cpu ")) { // Overall CPU stats (not per-core)
+ cpuLineFound = true;
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 11) {
+ // Full format with guest/guest_nice (kernel >= 2.6.24 with guest, >= 2.6.33 with guest_nice)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = Long.parseLong(parts[9]);
+ cpuGuestNice = Long.parseLong(parts[10]);
+ } else if (parts.length >= 9) {
+ // Format with steal but without guest/guest_nice (kernel >= 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ } else if (parts.length >= 8) {
+ // Older format without steal (kernel < 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = 0;
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ }
+ } else if (line.startsWith("ctxt ")) {
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 2) {
+ ctxt = Long.parseLong(parts[1]);
+ }
+ } else if (line.startsWith("procs_running ")) {
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 2) {
+ procsRunning = Long.parseLong(parts[1]);
+ }
+ } else if (line.startsWith("procs_blocked ")) {
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 2) {
+ procsBlocked = Long.parseLong(parts[1]);
+ }
+ }
+ }
+
+ // Validate that CPU line was found
+ if (!cpuLineFound) {
+ LOG.warn("failed to get /proc/stat: cpu line not found");
Review Comment:
Bug: The `processes` line from `/proc/stat` is never parsed. The code parses `ctxt`, `procs_running`, and `procs_blocked`, but is missing an `else if (line.startsWith("processes "))` block. As a result, the `processes` field always stays at `0`, and the `processesRate` (exposed as `process_forks_rate` metric) will always report `0.0`.
The test data file `stat_normal` contains `processes 5000000` on line 9, but this value is never read.
Suggested fix — add a block like:
```java
} else if (line.startsWith("processes ")) {
String[] parts = line.split("\\s+");
if (parts.length >= 2) {
processes = Long.parseLong(parts[1]);
}
}
```
between the `ctxt` and `procs_running` parsing blocks (or anywhere in the `while` loop).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on code in PR #60777:
URL: https://github.com/apache/doris/pull/60777#discussion_r2945860423
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -145,4 +209,140 @@ private void updateMemoryMetrics() {
}
}
+ private void updateCpuMetrics() {
+ String procFile = "/proc/stat";
+ if (FeConstants.runningUnitTest) {
+ procFile = getClass().getClassLoader().getResource(cpuStatTestFile).getFile();
+ }
+
+ try (FileReader fileReader = new FileReader(procFile);
+ BufferedReader br = new BufferedReader(fileReader)) {
+ String line;
+ boolean cpuLineFound = false;
+
+ // Store previous values for delta calculation
+ long prevTotal = prevCpuUser + prevCpuNice + prevCpuSystem + prevCpuIdle
+ + prevCpuIowait + prevCpuIrq + prevCpuSoftirq + prevCpuSteal;
+
+ while ((line = br.readLine()) != null) {
+ if (line.startsWith("cpu ")) { // Overall CPU stats (not per-core)
+ cpuLineFound = true;
+ String[] parts = WHITESPACE.split(line);
+ if (parts.length >= 11) {
+ // Full format with guest/guest_nice (kernel >= 2.6.24 with guest, >= 2.6.33 with guest_nice)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = Long.parseLong(parts[9]);
+ cpuGuestNice = Long.parseLong(parts[10]);
+ } else if (parts.length >= 9) {
+ // Format with steal but without guest/guest_nice (kernel >= 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ } else if (parts.length >= 8) {
+ // Older format without steal (kernel < 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = 0;
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ }
+ } else if (line.startsWith("ctxt ")) {
+ ctxt = parseSingleLongFromLine(line);
+ } else if (line.startsWith("processes ")) {
+ processes = parseSingleLongFromLine(line);
+ } else if (line.startsWith("procs_running ")) {
+ procsRunning = parseSingleLongFromLine(line);
+ } else if (line.startsWith("procs_blocked ")) {
+ procsBlocked = parseSingleLongFromLine(line);
+ }
+ }
+
+ // Validate that CPU line was found
+ if (!cpuLineFound) {
+ LOG.warn("failed to get /proc/stat: cpu line not found");
+ return;
+ }
+
+ // Calculate percentages based on delta (skip on first call to avoid inflated values)
+ long total = cpuUser + cpuNice + cpuSystem + cpuIdle + cpuIowait
+ + cpuIrq + cpuSoftirq + cpuSteal;
+
+ // Only calculate percentages if this is not the first call (prevTotal > 0)
+ if (prevTotal > 0) {
+ long delta = total - prevTotal;
+
+ if (delta > 0) {
+ // Calculate usage as percentage of non-idle time
+ long idleDelta = cpuIdle - prevCpuIdle;
+ long userDelta = cpuUser - prevCpuUser;
+ long systemDelta = cpuSystem - prevCpuSystem;
+ long iowaitDelta = cpuIowait - prevCpuIowait;
+ long stealDelta = cpuSteal - prevCpuSteal;
+
+ cpuUsagePercent = 100.0 * (delta - idleDelta) / delta;
+ cpuUserPercent = 100.0 * userDelta / delta;
+ cpuSystemPercent = 100.0 * systemDelta / delta;
+ cpuIowaitPercent = 100.0 * iowaitDelta / delta;
+ cpuStealPercent = 100.0 * stealDelta / delta;
+ }
+ }
+
+ // Store current CPU values as previous for next iteration
+ prevCpuUser = cpuUser;
+ prevCpuNice = cpuNice;
+ prevCpuSystem = cpuSystem;
+ prevCpuIdle = cpuIdle;
+ prevCpuIowait = cpuIowait;
+ prevCpuIrq = cpuIrq;
+ prevCpuSoftirq = cpuSoftirq;
+ prevCpuSteal = cpuSteal;
+
+ // Calculate context switch and process fork rates (skip on first call)
+ long currentTime = System.currentTimeMillis();
+ long timeDelta = currentTime - lastUpdateTime;
+
+ if (timeDelta > 0) {
+ double timeInSeconds = timeDelta / 1000.0;
+ if (prevCtxt > 0) {
+ ctxtRate = (ctxt - prevCtxt) / timeInSeconds;
+ }
+ if (prevProcesses > 0) {
+ processesRate = (processes - prevProcesses) / timeInSeconds;
+ }
+ }
+
+ // Store current values as previous for next iteration
+ prevCtxt = ctxt;
+ prevProcesses = processes;
+ lastUpdateTime = currentTime;
+
+ } catch (Exception e) {
+ LOG.warn("failed to get /proc/stat: {}", e.getMessage(), e);
+ }
+ }
+
+ private long parseSingleLongFromLine(String line) {
+ String[] parts = WHITESPACE.split(line);
Review Comment:
added a try-catch clause
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3906823966
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4017297550
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4017317548
## 1. [Critical] Split Unrelated Changes into Separate PRs
This PR bundles two completely unrelated changes:
- **CPU metrics enhancement** in `SystemMetrics` / `MetricRepo`
- **`ConcurrentModificationException` bug fix** in `PartitionCompensator`
Mixing unrelated changes in a single PR makes review harder, complicates git blame/bisect, and increases the risk of unintended regressions. These should be split into two independent PRs:
- PR A: `[Enhancement](metrics) Added CPU usage metrics`
- PR B: `[Fix](mv) Fix ConcurrentModificationException in PartitionCompensator`
---
## 2. [Medium] Improve Unit Test Coverage for CPU Percentage Calculation
The current `testCpuMetrics()` test cannot verify that the percentage calculation logic is correct. Since `MetricRepo.init()` calls `updateMetrics()` only once, it is always the "first call" — `prevTotal` is 0, so all percentage values default to 0.0. The assertion `>= 0.0` passes trivially but does not validate functional correctness.
**Suggestion**: call `update()` twice to establish a delta, then verify the percentage values against expected results computed from the test data.
```java
@Test
public void testCpuPercentageCalculation() {
SystemMetrics metrics = new SystemMetrics();
// First update establishes the baseline
metrics.update();
// Second update computes percentages based on delta
metrics.update();
// With static test data (stat_normal), both snapshots are identical,
// so delta = 0 and percentages remain 0.
// Ideally, use two different test data files to simulate a real delta scenario.
}
```
Also, `process_forks_rate` is asserted as `>= 0.0`, which always passes even though the value is always exactly 0 (due to the missing `processes` parsing). The test should validate the actual expected value after the parsing bug is fixed.
---
## 3. [Medium] Reduce Code Duplication in Line Parsing
The parsing logic for `ctxt`, `procs_running`, `procs_blocked` (and the missing `processes`) follows an identical pattern: split the line by whitespace, check length, and parse the second token. This can be extracted into a helper method:
```java
private long parseSingleLongFromLine(String line) {
String[] parts = line.split("\\s+");
return parts.length >= 2 ? Long.parseLong(parts[1]) : 0;
}
```
Usage:
```java
} else if (line.startsWith("ctxt ")) {
ctxt = parseSingleLongFromLine(line);
} else if (line.startsWith("processes ")) {
processes = parseSingleLongFromLine(line);
} else if (line.startsWith("procs_running ")) {
procsRunning = parseSingleLongFromLine(line);
} else if (line.startsWith("procs_blocked ")) {
procsBlocked = parseSingleLongFromLine(line);
}
```
---
## 4. [Low] Fix Rate Calculation Guard Condition
The current condition for calculating context switch and process fork rates is:
```java
if (timeDelta > 0 && prevCtxt > 0 && prevProcesses > 0) {
```
The `&&` relationship means that if **either** `prevCtxt` or `prevProcesses` is 0, **both** rate calculations are skipped. This is problematic because:
1. Even after fixing the `processes` parsing bug, `prevProcesses` will be 0 on the first call, which also blocks `ctxtRate` calculation.
2. `ctxtRate` and `processesRate` are independent — one should not prevent the other.
**Suggested fix** — either use separate guards or use a general "is first call" flag:
```diff
-if (timeDelta > 0 && prevCtxt > 0 && prevProcesses > 0) {
- double timeInSeconds = timeDelta / 1000.0;
- ctxtRate = (ctxt - prevCtxt) / timeInSeconds;
- processesRate = (processes - prevProcesses) / timeInSeconds;
-}
+if (timeDelta > 0) {
+ double timeInSeconds = timeDelta / 1000.0;
+ if (prevCtxt > 0) {
+ ctxtRate = (ctxt - prevCtxt) / timeInSeconds;
+ }
+ if (prevProcesses > 0) {
+ processesRate = (processes - prevProcesses) / timeInSeconds;
+ }
+}
```
---
## 5. [Low] Consider Using Getter Methods for Field Access
All new fields in `SystemMetrics` are `protected` and accessed directly from anonymous `GaugeMetric` classes in `MetricRepo` (e.g., `SYSTEM_METRICS.cpuUsagePercent`). Using getter methods would improve encapsulation and maintainability:
```diff
-return SYSTEM_METRICS.cpuUsagePercent;
+return SYSTEM_METRICS.getCpuUsagePercent();
```
> **Note**: The existing TCP and memory metrics also use direct `protected` field access, so maintaining consistency with the current codebase style is also a valid choice.
---
## 6. [Low] Pre-compile Regex Pattern
`line.split("\\s+")` is called multiple times inside a loop. Each call internally compiles the regex pattern. While the performance impact is negligible for this use case, pre-compiling the pattern is a Java best practice:
```java
private static final Pattern WHITESPACE = Pattern.compile("\\s+");
// Usage:
String[] parts = WHITESPACE.split(line);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3991669251
<details>
<summary>TPC-DS: <b>Total hot run time: 184029 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd97d3766cb5e37a343512341fc3dca6b35afd4c, data reload: false
query5 4808 648 549 549
query6 358 219 208 208
query7 4253 484 275 275
query8 336 255 243 243
query9 8790 2781 2732 2732
query10 578 383 345 345
query11 16934 17540 17424 17424
query12 203 152 136 136
query13 1457 494 354 354
query14 7742 3417 3350 3350
query14_1 2875 2922 2870 2870
query15 216 197 174 174
query16 1003 486 483 483
query17 1077 794 607 607
query18 2706 439 341 341
query19 212 202 175 175
query20 131 127 127 127
query21 214 136 115 115
query22 4758 5014 4687 4687
query23 17176 16827 16587 16587
query23_1 16666 16768 16604 16604
query24 7000 1617 1236 1236
query24_1 1248 1246 1239 1239
query25 561 487 448 448
query26 1242 266 152 152
query27 2772 465 293 293
query28 4497 1927 1887 1887
query29 823 581 491 491
query30 319 248 209 209
query31 886 745 665 665
query32 90 79 73 73
query33 527 350 297 297
query34 922 954 597 597
query35 651 721 591 591
query36 1130 1127 900 900
query37 153 106 83 83
query38 2964 2970 2866 2866
query39 885 866 837 837
query39_1 857 826 830 826
query40 236 159 137 137
query41 85 65 66 65
query42 108 104 108 104
query43 390 377 351 351
query44
query45 204 196 186 186
query46 891 991 608 608
query47 2162 2168 2055 2055
query48 324 319 233 233
query49 655 481 393 393
query50 730 298 241 241
query51 4070 4076 4073 4073
query52 107 108 103 103
query53 294 342 299 299
query54 317 288 280 280
query55 134 86 81 81
query56 303 311 309 309
query57 1376 1346 1287 1287
query58 291 281 277 277
query59 2562 2709 2540 2540
query60 340 338 316 316
query61 146 146 147 146
query62 609 573 524 524
query63 312 273 277 273
query64 4782 1244 969 969
query65
query66 1380 457 371 371
query67 16431 16459 16327 16327
query68
query69 400 325 284 284
query70 991 926 963 926
query71 338 305 302 302
query72 2810 2664 2422 2422
query73 560 550 337 337
query74 9989 9986 9798 9798
query75 2868 2770 2471 2471
query76 2294 1030 669 669
query77 386 389 316 316
query78 11239 11525 10649 10649
query79 2064 816 616 616
query80 1556 634 545 545
query81 567 279 251 251
query82 1034 151 116 116
query83 357 263 245 245
query84 256 122 107 107
query85 938 479 418 418
query86 413 339 314 314
query87 3130 3103 2963 2963
query88 3635 2690 2680 2680
query89 438 367 345 345
query90 1981 176 161 161
query91 167 161 134 134
query92 82 81 73 73
query93 1116 834 519 519
query94 633 276 301 276
query95 585 391 319 319
query96 647 528 228 228
query97 2462 2489 2381 2381
query98 234 220 212 212
query99 1034 1016 921 921
Total cold run time: 256433 ms
Total hot run time: 184029 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3995740149
# FE Regression Coverage Report
Increment line coverage ` 63.69% (114/179)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3989812639
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3992840132
# FE Regression Coverage Report
Increment line coverage ` 62.43% (108/173)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60777_bd97d3766cb5e37a343512341fc3dca6b35afd4c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60777_bd97d3766cb5e37a343512341fc3dca6b35afd4c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-5706482965
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4074299212
<details>
<summary>TPC-DS: <b>Total hot run time: 168254 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d08356646505426a6247889bb0e181e9a775a453, data reload: false
query5 4339 631 490 490
query6 342 235 213 213
query7 4224 460 269 269
query8 349 250 237 237
query9 8731 2764 2761 2761
query10 519 405 330 330
query11 6972 5057 4885 4885
query12 189 124 125 124
query13 1254 472 336 336
query14 5727 3689 3471 3471
query14_1 2856 2836 2752 2752
query15 201 189 172 172
query16 971 478 435 435
query17 865 715 639 639
query18 2447 455 359 359
query19 224 217 193 193
query20 141 131 128 128
query21 216 133 111 111
query22 13159 14106 14847 14106
query23 16319 15889 15587 15587
query23_1 16254 15747 15478 15478
query24 7175 1629 1221 1221
query24_1 1249 1238 1224 1224
query25 581 497 449 449
query26 1237 265 158 158
query27 2757 496 302 302
query28 4485 1884 1898 1884
query29 884 591 525 525
query30 313 234 197 197
query31 1023 957 895 895
query32 85 75 74 74
query33 536 352 297 297
query34 901 872 540 540
query35 676 702 626 626
query36 1070 1152 988 988
query37 135 100 85 85
query38 2951 2902 2858 2858
query39 874 821 825 821
query39_1 815 794 785 785
query40 244 162 139 139
query41 71 107 58 58
query42 258 254 253 253
query43 236 236 219 219
query44
query45 196 191 184 184
query46 882 983 609 609
query47 2125 2130 2105 2105
query48 306 312 229 229
query49 643 463 379 379
query50 692 278 209 209
query51 4172 4027 3961 3961
query52 264 263 254 254
query53 293 341 287 287
query54 303 273 265 265
query55 94 85 82 82
query56 318 323 317 317
query57 1916 1644 1549 1549
query58 282 278 276 276
query59 2780 2939 2735 2735
query60 344 346 326 326
query61 154 155 153 153
query62 618 588 542 542
query63 307 285 277 277
query64 5096 1326 1021 1021
query65
query66 1451 462 357 357
query67 24220 24200 24055 24055
query68
query69 409 318 293 293
query70 909 986 969 969
query71 344 319 300 300
query72 2856 2711 2427 2427
query73 543 545 322 322
query74 9645 9541 9476 9476
query75 2849 2746 2461 2461
query76 2297 1023 688 688
query77 370 406 310 310
query78 10932 11166 10460 10460
query79 2526 766 606 606
query80 1787 635 550 550
query81 562 261 230 230
query82 1001 155 118 118
query83 342 277 249 249
query84 296 120 105 105
query85 914 524 454 454
query86 434 314 285 285
query87 3159 3081 2995 2995
query88 3571 2704 2650 2650
query89 428 374 341 341
query90 2045 185 176 176
query91 167 163 140 140
query92 79 72 67 67
query93 1104 873 496 496
query94 639 322 297 297
query95 592 392 317 317
query96 648 510 227 227
query97 2503 2490 2407 2407
query98 242 228 219 219
query99 987 1013 915 915
Total cold run time: 251988 ms
Total hot run time: 168254 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3995257963
# FE UT Coverage Report
Increment line coverage ` 70.95% (127/179)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3996454380
<details>
<summary>TPC-DS: <b>Total hot run time: 184127 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d6a691021c9aea4150fa2352fad9346ebcbc5e9, data reload: false
query5 4363 639 528 528
query6 354 235 211 211
query7 4217 475 286 286
query8 358 247 245 245
query9 8696 2796 2800 2796
query10 507 381 336 336
query11 17005 17752 17216 17216
query12 209 146 140 140
query13 1492 481 383 383
query14 6910 3520 3115 3115
query14_1 2988 3002 2937 2937
query15 209 202 207 202
query16 1005 492 501 492
query17 1248 846 640 640
query18 2854 474 365 365
query19 219 218 185 185
query20 146 132 144 132
query21 213 134 116 116
query22 5370 5428 4585 4585
query23 17144 16756 16613 16613
query23_1 16619 16611 16756 16611
query24 7136 1617 1231 1231
query24_1 1246 1254 1235 1235
query25 566 482 461 461
query26 1239 261 156 156
query27 2758 463 281 281
query28 4477 1880 1892 1880
query29 805 578 469 469
query30 309 250 211 211
query31 870 731 663 663
query32 77 74 70 70
query33 522 355 283 283
query34 929 934 563 563
query35 626 680 606 606
query36 1093 1133 954 954
query37 136 97 87 87
query38 2965 2905 2862 2862
query39 901 855 837 837
query39_1 870 837 826 826
query40 226 149 136 136
query41 61 63 58 58
query42 107 102 102 102
query43 378 394 350 350
query44
query45 210 192 183 183
query46 879 989 614 614
query47 2088 2109 2014 2014
query48 308 309 231 231
query49 638 463 425 425
query50 704 281 210 210
query51 4120 4052 4106 4052
query52 104 108 94 94
query53 294 342 300 300
query54 295 266 269 266
query55 91 83 83 83
query56 339 319 308 308
query57 1369 1335 1269 1269
query58 288 287 280 280
query59 2605 2735 2558 2558
query60 343 327 324 324
query61 148 150 150 150
query62 637 581 558 558
query63 314 279 275 275
query64 4876 1277 963 963
query65
query66 1389 453 345 345
query67 16341 16421 16397 16397
query68
query69 399 308 277 277
query70 1030 972 989 972
query71 332 302 293 293
query72 2826 2613 2590 2590
query73 536 554 316 316
query74 9942 9999 9768 9768
query75 2836 2770 2502 2502
query76 2308 1042 682 682
query77 370 381 309 309
query78 11124 11408 10681 10681
query79 2435 798 621 621
query80 1680 631 554 554
query81 571 280 249 249
query82 698 148 113 113
query83 353 265 240 240
query84 252 118 96 96
query85 885 493 453 453
query86 416 321 303 303
query87 3077 3078 2989 2989
query88 3585 2670 2674 2670
query89 434 373 344 344
query90 2045 181 173 173
query91 164 159 128 128
query92 79 76 68 68
query93 1103 853 508 508
query94 623 328 300 300
query95 589 414 309 309
query96 642 507 232 232
query97 2455 2465 2399 2399
query98 228 242 221 221
query99 981 1004 913 913
Total cold run time: 255632 ms
Total hot run time: 184127 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4070235935
hi @sahildevgon , just serval minor issues mentioned by AI review, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4024255818
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4025253192
<details>
<summary>TPC-DS: <b>Total hot run time: 153416 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dbe88266d544f100cb756937c5352c62a544e1ba, data reload: false
query5 4344 645 525 525
query6 326 221 192 192
query7 4242 472 274 274
query8 346 259 244 244
query9 8713 2732 2694 2694
query10 525 374 352 352
query11 7402 5839 5798 5798
query12 191 132 128 128
query13 1288 452 345 345
query14 5601 3869 3599 3599
query14_1 2807 2800 2810 2800
query15 206 197 181 181
query16 975 464 453 453
query17 895 722 624 624
query18 2447 446 347 347
query19 221 213 190 190
query20 139 134 130 130
query21 226 147 121 121
query22 4778 5088 4726 4726
query23 16564 16032 15581 15581
query23_1 15927 15966 15876 15876
query24 7604 1849 1292 1292
query24_1 1245 1267 1295 1267
query25 570 493 436 436
query26 1356 290 153 153
query27 2739 478 282 282
query28 4522 1866 1868 1866
query29 845 601 482 482
query30 312 237 217 217
query31 1367 1298 1228 1228
query32 77 79 71 71
query33 516 332 277 277
query34 898 911 566 566
query35 658 676 616 616
query36 1114 1121 986 986
query37 137 91 83 83
query38 2939 2911 2882 2882
query39 884 873 848 848
query39_1 825 842 814 814
query40 232 156 134 134
query41 63 65 57 57
query42 301 298 299 298
query43 244 249 218 218
query44
query45 198 196 192 192
query46 881 985 611 611
query47 2129 2142 2044 2044
query48 317 319 221 221
query49 619 483 382 382
query50 668 273 210 210
query51 4167 4116 4052 4052
query52 294 292 279 279
query53 298 371 280 280
query54 304 283 268 268
query55 93 86 87 86
query56 318 321 306 306
query57 1365 1326 1274 1274
query58 295 286 275 275
query59 1345 1433 1265 1265
query60 352 336 308 308
query61 155 146 163 146
query62 631 598 548 548
query63 308 285 281 281
query64 5061 1273 1003 1003
query65
query66 1463 465 351 351
query67 16543 16468 16380 16380
query68
query69 390 315 284 284
query70 1011 1003 965 965
query71 346 311 294 294
query72 2811 2746 2577 2577
query73 568 547 326 326
query74 9979 9932 9828 9828
query75 2867 2790 2493 2493
query76 2290 1038 670 670
query77 372 389 330 330
query78 11146 11249 10634 10634
query79 3124 781 581 581
query80 1761 671 577 577
query81 601 284 246 246
query82 999 151 126 126
query83 378 260 238 238
query84 298 119 103 103
query85 933 489 432 432
query86 438 321 315 315
query87 3125 3107 3037 3037
query88 3542 2631 2626 2626
query89 417 372 352 352
query90 1978 175 171 171
query91 169 159 136 136
query92 78 79 71 71
query93 1358 826 519 519
query94 655 314 275 275
query95 577 347 388 347
query96 647 512 225 225
query97 2468 2483 2433 2433
query98 242 246 220 220
query99 996 981 913 913
Total cold run time: 237227 ms
Total hot run time: 153416 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3996187120
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3996934868
# FE Regression Coverage Report
Increment line coverage ` 63.69% (114/179)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #60777: [Enhancement](metrics)Added CPU usage metrics
URL: https://github.com/apache/doris/pull/60777
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4025166818
<details>
<summary>TPC-H: <b>Total hot run time: 27545 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dbe88266d544f100cb756937c5352c62a544e1ba, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17734 4623 4296 4296
q2 q3 10649 773 504 504
q4 4676 354 251 251
q5 7568 1203 1008 1008
q6 175 173 145 145
q7 813 830 682 682
q8 9643 1431 1320 1320
q9 5250 4754 4642 4642
q10 6314 1905 1653 1653
q11 455 251 240 240
q12 758 567 474 474
q13 18045 2946 2197 2197
q14 234 224 212 212
q15 925 804 798 798
q16 760 727 681 681
q17 747 884 398 398
q18 5979 5397 5282 5282
q19 1542 992 590 590
q20 503 486 395 395
q21 4602 2033 1482 1482
q22 389 361 295 295
Total cold run time: 97761 ms
Total hot run time: 27545 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4771 4571 4615 4571
q2 q3 3888 4420 3817 3817
q4 859 1178 809 809
q5 4038 4432 4316 4316
q6 180 172 142 142
q7 1769 1654 1543 1543
q8 2581 2675 2549 2549
q9 7398 7386 7587 7386
q10 3761 3971 3560 3560
q11 488 438 409 409
q12 482 593 448 448
q13 2719 3327 2320 2320
q14 297 304 287 287
q15 881 852 827 827
q16 709 758 707 707
q17 1139 1397 1383 1383
q18 7271 6922 6660 6660
q19 936 861 859 859
q20 2048 2197 1985 1985
q21 4003 3425 3426 3425
q22 484 442 388 388
Total cold run time: 50702 ms
Total hot run time: 48391 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4101394346
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on code in PR #60777:
URL: https://github.com/apache/doris/pull/60777#discussion_r2945857783
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -145,4 +209,140 @@ private void updateMemoryMetrics() {
}
}
+ private void updateCpuMetrics() {
+ String procFile = "/proc/stat";
+ if (FeConstants.runningUnitTest) {
+ procFile = getClass().getClassLoader().getResource(cpuStatTestFile).getFile();
+ }
+
+ try (FileReader fileReader = new FileReader(procFile);
+ BufferedReader br = new BufferedReader(fileReader)) {
+ String line;
+ boolean cpuLineFound = false;
+
+ // Store previous values for delta calculation
+ long prevTotal = prevCpuUser + prevCpuNice + prevCpuSystem + prevCpuIdle
+ + prevCpuIowait + prevCpuIrq + prevCpuSoftirq + prevCpuSteal;
+
+ while ((line = br.readLine()) != null) {
+ if (line.startsWith("cpu ")) { // Overall CPU stats (not per-core)
+ cpuLineFound = true;
+ String[] parts = WHITESPACE.split(line);
Review Comment:
Added an else clause
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3995154419
<details>
<summary>TPC-DS: <b>Total hot run time: 183129 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d6a691021c9aea4150fa2352fad9346ebcbc5e9, data reload: false
query5 5819 664 547 547
query6 330 231 207 207
query7 4234 490 299 299
query8 346 247 241 241
query9 8738 2812 2773 2773
query10 595 387 357 357
query11 17088 16880 16595 16595
query12 200 129 128 128
query13 1288 465 359 359
query14 7885 3258 3006 3006
query14_1 2881 2904 2864 2864
query15 209 200 179 179
query16 992 456 451 451
query17 1312 728 617 617
query18 2716 460 347 347
query19 234 216 187 187
query20 146 131 134 131
query21 234 147 123 123
query22 4929 4899 4722 4722
query23 17284 16814 16514 16514
query23_1 16744 16703 16821 16703
query24 6752 1629 1248 1248
query24_1 1257 1274 1266 1266
query25 562 496 430 430
query26 1247 276 157 157
query27 2728 489 296 296
query28 4531 1919 1903 1903
query29 837 619 502 502
query30 319 254 215 215
query31 885 740 665 665
query32 102 79 77 77
query33 531 355 297 297
query34 913 938 578 578
query35 650 705 597 597
query36 1069 1120 1006 1006
query37 142 106 90 90
query38 2991 2958 2825 2825
query39 913 883 854 854
query39_1 851 826 831 826
query40 236 167 183 167
query41 74 62 63 62
query42 112 109 107 107
query43 390 386 359 359
query44
query45 198 186 183 183
query46 913 982 616 616
query47 2159 2118 2070 2070
query48 331 325 248 248
query49 632 486 394 394
query50 731 281 219 219
query51 4109 4087 4080 4080
query52 108 110 101 101
query53 297 352 292 292
query54 301 282 267 267
query55 86 86 82 82
query56 317 306 313 306
query57 1382 1349 1283 1283
query58 303 282 277 277
query59 2548 2675 2474 2474
query60 344 346 332 332
query61 156 153 149 149
query62 630 590 536 536
query63 318 283 311 283
query64 4806 1284 978 978
query65
query66 1381 464 366 366
query67 16438 16456 16294 16294
query68
query69 412 316 293 293
query70 1014 1000 979 979
query71 358 321 314 314
query72 2824 2656 2397 2397
query73 557 564 335 335
query74 10011 9916 9738 9738
query75 2860 2736 2467 2467
query76 2261 1087 704 704
query77 375 384 321 321
query78 11336 11450 10632 10632
query79 1167 799 613 613
query80 1640 658 550 550
query81 576 277 245 245
query82 996 151 118 118
query83 350 270 243 243
query84 247 118 100 100
query85 1340 480 436 436
query86 484 304 292 292
query87 3123 3145 3028 3028
query88 3631 2679 2657 2657
query89 469 381 357 357
query90 1883 178 170 170
query91 172 160 137 137
query92 80 88 82 82
query93 1295 864 509 509
query94 656 345 310 310
query95 567 332 315 315
query96 646 539 234 234
query97 2444 2480 2415 2415
query98 235 216 223 216
query99 969 987 946 946
Total cold run time: 257472 ms
Total hot run time: 183129 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60777:
URL: https://github.com/apache/doris/pull/60777#discussion_r2942015245
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -54,10 +56,72 @@ public class SystemMetrics {
protected long buffers = 0;
// Memory in the pagecache (Diskcache and Shared Memory)
protected long cached = 0;
+ // CPU time metrics (in USER_HZ, typically 1/100th of a second)
+ // Time spent in user mode
+ protected long cpuUser = 0;
+ // Time spent in user mode with low priority (nice)
+ protected long cpuNice = 0;
+ // Time spent in system/kernel mode
+ protected long cpuSystem = 0;
+ // Time spent idle
+ protected long cpuIdle = 0;
+ // Time waiting for I/O to complete
+ protected long cpuIowait = 0;
+ // Time servicing hardware interrupts
+ protected long cpuIrq = 0;
+ // Time servicing software interrupts
+ protected long cpuSoftirq = 0;
+ // Time stolen by hypervisor (for VMs)
+ protected long cpuSteal = 0;
+ // Time spent running virtual CPU for guest OS
+ protected long cpuGuest = 0;
+ // Time spent running niced guest
+ protected long cpuGuestNice = 0;
+
+ // Package-private: allows tests to override the /proc/stat resource path
+ String cpuStatTestFile = "data/stat_normal";
+
Review Comment:
**[Suggestion]** This package-private mutable field exists solely for test injection. It leaks test infrastructure into the production class. Consider using a `protected` method that returns the file path (which tests can override), or a constructor parameter, to keep test concerns separated from production code.
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -145,4 +209,140 @@ private void updateMemoryMetrics() {
}
}
+ private void updateCpuMetrics() {
+ String procFile = "/proc/stat";
+ if (FeConstants.runningUnitTest) {
+ procFile = getClass().getClassLoader().getResource(cpuStatTestFile).getFile();
+ }
+
+ try (FileReader fileReader = new FileReader(procFile);
+ BufferedReader br = new BufferedReader(fileReader)) {
+ String line;
+ boolean cpuLineFound = false;
+
+ // Store previous values for delta calculation
+ long prevTotal = prevCpuUser + prevCpuNice + prevCpuSystem + prevCpuIdle
+ + prevCpuIowait + prevCpuIrq + prevCpuSoftirq + prevCpuSteal;
+
+ while ((line = br.readLine()) != null) {
+ if (line.startsWith("cpu ")) { // Overall CPU stats (not per-core)
+ cpuLineFound = true;
+ String[] parts = WHITESPACE.split(line);
Review Comment:
**[Robustness]** When `parts.length < 8` (malformed `cpu` line), `cpuLineFound` is set to `true` here but none of the if/else-if branches execute, so no CPU fields are updated. The code then proceeds to the percentage calculation at line 290 using stale/zero values, which silently produces incorrect metrics.
Consider adding an `else` clause that either logs a warning and sets `cpuLineFound = false`, or explicitly handles the malformed case:
```java
} else {
LOG.warn("unexpected /proc/stat cpu line format, parts.length={}: {}", parts.length, line);
cpuLineFound = false;
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -145,4 +209,140 @@ private void updateMemoryMetrics() {
}
}
+ private void updateCpuMetrics() {
+ String procFile = "/proc/stat";
+ if (FeConstants.runningUnitTest) {
+ procFile = getClass().getClassLoader().getResource(cpuStatTestFile).getFile();
+ }
+
+ try (FileReader fileReader = new FileReader(procFile);
+ BufferedReader br = new BufferedReader(fileReader)) {
+ String line;
+ boolean cpuLineFound = false;
+
+ // Store previous values for delta calculation
+ long prevTotal = prevCpuUser + prevCpuNice + prevCpuSystem + prevCpuIdle
+ + prevCpuIowait + prevCpuIrq + prevCpuSoftirq + prevCpuSteal;
+
+ while ((line = br.readLine()) != null) {
+ if (line.startsWith("cpu ")) { // Overall CPU stats (not per-core)
+ cpuLineFound = true;
+ String[] parts = WHITESPACE.split(line);
+ if (parts.length >= 11) {
+ // Full format with guest/guest_nice (kernel >= 2.6.24 with guest, >= 2.6.33 with guest_nice)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = Long.parseLong(parts[9]);
+ cpuGuestNice = Long.parseLong(parts[10]);
+ } else if (parts.length >= 9) {
+ // Format with steal but without guest/guest_nice (kernel >= 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ } else if (parts.length >= 8) {
+ // Older format without steal (kernel < 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = 0;
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ }
+ } else if (line.startsWith("ctxt ")) {
+ ctxt = parseSingleLongFromLine(line);
+ } else if (line.startsWith("processes ")) {
+ processes = parseSingleLongFromLine(line);
+ } else if (line.startsWith("procs_running ")) {
+ procsRunning = parseSingleLongFromLine(line);
+ } else if (line.startsWith("procs_blocked ")) {
+ procsBlocked = parseSingleLongFromLine(line);
+ }
+ }
+
+ // Validate that CPU line was found
+ if (!cpuLineFound) {
+ LOG.warn("failed to get /proc/stat: cpu line not found");
+ return;
+ }
+
+ // Calculate percentages based on delta (skip on first call to avoid inflated values)
+ long total = cpuUser + cpuNice + cpuSystem + cpuIdle + cpuIowait
+ + cpuIrq + cpuSoftirq + cpuSteal;
+
+ // Only calculate percentages if this is not the first call (prevTotal > 0)
+ if (prevTotal > 0) {
+ long delta = total - prevTotal;
+
+ if (delta > 0) {
+ // Calculate usage as percentage of non-idle time
+ long idleDelta = cpuIdle - prevCpuIdle;
+ long userDelta = cpuUser - prevCpuUser;
+ long systemDelta = cpuSystem - prevCpuSystem;
+ long iowaitDelta = cpuIowait - prevCpuIowait;
+ long stealDelta = cpuSteal - prevCpuSteal;
+
+ cpuUsagePercent = 100.0 * (delta - idleDelta) / delta;
+ cpuUserPercent = 100.0 * userDelta / delta;
+ cpuSystemPercent = 100.0 * systemDelta / delta;
+ cpuIowaitPercent = 100.0 * iowaitDelta / delta;
+ cpuStealPercent = 100.0 * stealDelta / delta;
+ }
+ }
+
+ // Store current CPU values as previous for next iteration
+ prevCpuUser = cpuUser;
+ prevCpuNice = cpuNice;
+ prevCpuSystem = cpuSystem;
+ prevCpuIdle = cpuIdle;
+ prevCpuIowait = cpuIowait;
+ prevCpuIrq = cpuIrq;
+ prevCpuSoftirq = cpuSoftirq;
+ prevCpuSteal = cpuSteal;
+
+ // Calculate context switch and process fork rates (skip on first call)
+ long currentTime = System.currentTimeMillis();
+ long timeDelta = currentTime - lastUpdateTime;
+
+ if (timeDelta > 0) {
+ double timeInSeconds = timeDelta / 1000.0;
+ if (prevCtxt > 0) {
+ ctxtRate = (ctxt - prevCtxt) / timeInSeconds;
+ }
+ if (prevProcesses > 0) {
+ processesRate = (processes - prevProcesses) / timeInSeconds;
+ }
+ }
+
+ // Store current values as previous for next iteration
+ prevCtxt = ctxt;
+ prevProcesses = processes;
+ lastUpdateTime = currentTime;
+
+ } catch (Exception e) {
+ LOG.warn("failed to get /proc/stat: {}", e.getMessage(), e);
+ }
+ }
+
+ private long parseSingleLongFromLine(String line) {
+ String[] parts = WHITESPACE.split(line);
Review Comment:
**[Robustness]** `Long.parseLong(parts[1])` can throw `NumberFormatException` if the value is malformed. While the outer `catch (Exception e)` block will handle it, the exception short-circuits the rest of `updateCpuMetrics()`, skipping the updates to `prevCtxt`, `prevProcesses`, and `lastUpdateTime` (lines 325-336). On the next successful invocation, the stale `prev*` values combined with the new `lastUpdateTime` delta will produce a spurious spike in `ctxtRate` and `processesRate`.
Consider wrapping the parse in a try-catch or adding validation:
```java
private long parseSingleLongFromLine(String line) {
String[] parts = WHITESPACE.split(line);
if (parts.length >= 2) {
try {
return Long.parseLong(parts[1]);
} catch (NumberFormatException e) {
LOG.warn("failed to parse long from line: {}", line);
}
}
return 0;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3991163085
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on code in PR #60777:
URL: https://github.com/apache/doris/pull/60777#discussion_r2945867750
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -54,10 +56,72 @@ public class SystemMetrics {
protected long buffers = 0;
// Memory in the pagecache (Diskcache and Shared Memory)
protected long cached = 0;
+ // CPU time metrics (in USER_HZ, typically 1/100th of a second)
+ // Time spent in user mode
+ protected long cpuUser = 0;
+ // Time spent in user mode with low priority (nice)
+ protected long cpuNice = 0;
+ // Time spent in system/kernel mode
+ protected long cpuSystem = 0;
+ // Time spent idle
+ protected long cpuIdle = 0;
+ // Time waiting for I/O to complete
+ protected long cpuIowait = 0;
+ // Time servicing hardware interrupts
+ protected long cpuIrq = 0;
+ // Time servicing software interrupts
+ protected long cpuSoftirq = 0;
+ // Time stolen by hypervisor (for VMs)
+ protected long cpuSteal = 0;
+ // Time spent running virtual CPU for guest OS
+ protected long cpuGuest = 0;
+ // Time spent running niced guest
+ protected long cpuGuestNice = 0;
+
+ // Package-private: allows tests to override the /proc/stat resource path
+ String cpuStatTestFile = "data/stat_normal";
+
Review Comment:
added a protected method for path(Overridable in tests)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4074233819
<details>
<summary>TPC-H: <b>Total hot run time: 26847 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d08356646505426a6247889bb0e181e9a775a453, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17670 4498 4333 4333
q2 q3 10645 761 530 530
q4 4676 352 249 249
q5 7662 1212 1036 1036
q6 176 186 145 145
q7 790 845 675 675
q8 10263 1446 1274 1274
q9 5607 4624 4710 4624
q10 6321 1925 1645 1645
q11 475 251 232 232
q12 748 578 464 464
q13 18044 2947 2215 2215
q14 231 227 219 219
q15 q16 737 740 661 661
q17 724 833 454 454
q18 5864 5498 5241 5241
q19 1494 977 612 612
q20 540 479 369 369
q21 4519 1940 1581 1581
q22 416 350 288 288
Total cold run time: 97602 ms
Total hot run time: 26847 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4774 4591 4682 4591
q2 q3 3919 4343 3821 3821
q4 905 1207 801 801
q5 4044 4378 4428 4378
q6 189 184 147 147
q7 1771 1639 1536 1536
q8 2474 2772 2593 2593
q9 7542 7443 7435 7435
q10 3751 4011 3603 3603
q11 526 432 417 417
q12 491 591 450 450
q13 2716 3496 2341 2341
q14 288 308 282 282
q15 q16 734 824 732 732
q17 1175 1432 1435 1432
q18 7267 6780 6602 6602
q19 942 874 1004 874
q20 2143 2160 1979 1979
q21 4011 3491 3321 3321
q22 471 424 389 389
Total cold run time: 50133 ms
Total hot run time: 47724 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4074445947
# FE UT Coverage Report
Increment line coverage ` 79.89% (143/179)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60777_d08356646505426a6247889bb0e181e9a775a453/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60777_d08356646505426a6247889bb0e181e9a775a453/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3995028748
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3991602662
<details>
<summary>TPC-H: <b>Total hot run time: 28998 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd97d3766cb5e37a343512341fc3dca6b35afd4c, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17593 4484 4299 4299
q2 q3 10643 771 542 542
q4 4675 363 255 255
q5 7544 1229 1025 1025
q6 177 177 147 147
q7 803 841 687 687
q8 9282 1482 1324 1324
q9 4789 4780 4701 4701
q10 6795 1861 1660 1660
q11 458 260 245 245
q12 705 567 468 468
q13 17774 4255 3439 3439
q14 235 234 212 212
q15 945 797 790 790
q16 758 718 677 677
q17 733 854 422 422
q18 6003 5317 5348 5317
q19 1240 973 618 618
q20 515 485 387 387
q21 4750 2030 1503 1503
q22 364 321 280 280
Total cold run time: 96781 ms
Total hot run time: 28998 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4701 4576 4533 4533
q2 q3 1827 2199 1766 1766
q4 866 1181 790 790
q5 4068 4423 4293 4293
q6 190 175 139 139
q7 1802 1658 1524 1524
q8 2596 2701 2563 2563
q9 7701 7366 7463 7366
q10 2698 2908 2455 2455
q11 541 438 418 418
q12 534 613 438 438
q13 3962 4490 3540 3540
q14 289 297 291 291
q15 863 813 778 778
q16 708 758 726 726
q17 1176 1565 1313 1313
q18 7071 6765 6571 6571
q19 909 898 949 898
q20 2149 2156 2000 2000
q21 3988 3495 3428 3428
q22 508 448 388 388
Total cold run time: 49147 ms
Total hot run time: 46218 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3997064369
# FE Regression Coverage Report
Increment line coverage ` 63.69% (114/179)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60777_8d6a691021c9aea4150fa2352fad9346ebcbc5e9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4009887155
Hi @morningman can you help with getting this pr reviewed and merged. Looking forward to same. Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on code in PR #60777:
URL: https://github.com/apache/doris/pull/60777#discussion_r2905852877
##########
fe/fe-core/src/main/java/org/apache/doris/metric/SystemMetrics.java:
##########
@@ -145,4 +204,139 @@ private void updateMemoryMetrics() {
}
}
+ private void updateCpuMetrics() {
+ String procFile = "/proc/stat";
+ if (FeConstants.runningUnitTest) {
+ procFile = getClass().getClassLoader().getResource("data/stat_normal").getFile();
+ }
+
+ try (FileReader fileReader = new FileReader(procFile);
+ BufferedReader br = new BufferedReader(fileReader)) {
+ String line;
+ boolean cpuLineFound = false;
+
+ // Store previous values for delta calculation
+ long prevTotal = prevCpuUser + prevCpuNice + prevCpuSystem + prevCpuIdle
+ + prevCpuIowait + prevCpuIrq + prevCpuSoftirq + prevCpuSteal;
+
+ while ((line = br.readLine()) != null) {
+ if (line.startsWith("cpu ")) { // Overall CPU stats (not per-core)
+ cpuLineFound = true;
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 11) {
+ // Full format with guest/guest_nice (kernel >= 2.6.24 with guest, >= 2.6.33 with guest_nice)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = Long.parseLong(parts[9]);
+ cpuGuestNice = Long.parseLong(parts[10]);
+ } else if (parts.length >= 9) {
+ // Format with steal but without guest/guest_nice (kernel >= 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = Long.parseLong(parts[8]);
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ } else if (parts.length >= 8) {
+ // Older format without steal (kernel < 2.6.11)
+ cpuUser = Long.parseLong(parts[1]);
+ cpuNice = Long.parseLong(parts[2]);
+ cpuSystem = Long.parseLong(parts[3]);
+ cpuIdle = Long.parseLong(parts[4]);
+ cpuIowait = Long.parseLong(parts[5]);
+ cpuIrq = Long.parseLong(parts[6]);
+ cpuSoftirq = Long.parseLong(parts[7]);
+ cpuSteal = 0;
+ cpuGuest = 0;
+ cpuGuestNice = 0;
+ }
+ } else if (line.startsWith("ctxt ")) {
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 2) {
+ ctxt = Long.parseLong(parts[1]);
+ }
+ } else if (line.startsWith("procs_running ")) {
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 2) {
+ procsRunning = Long.parseLong(parts[1]);
+ }
+ } else if (line.startsWith("procs_blocked ")) {
+ String[] parts = line.split("\\s+");
+ if (parts.length >= 2) {
+ procsBlocked = Long.parseLong(parts[1]);
+ }
+ }
+ }
+
+ // Validate that CPU line was found
+ if (!cpuLineFound) {
+ LOG.warn("failed to get /proc/stat: cpu line not found");
Review Comment:
Fixed , added the block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-4101394226
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3989632495
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "sahildevgon (via GitHub)" <gi...@apache.org>.
sahildevgon commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3985212163
Hi @morningman , just following up on this. Let me know if I can do something better to get this merged soon. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](metrics)Added CPU usage metrics [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60777:
URL: https://github.com/apache/doris/pull/60777#issuecomment-3996393228
<details>
<summary>TPC-H: <b>Total hot run time: 28711 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d6a691021c9aea4150fa2352fad9346ebcbc5e9, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17656 4590 4299 4299
q2 q3 10640 770 515 515
q4 4679 352 256 256
q5 7541 1187 1020 1020
q6 173 175 148 148
q7 764 838 665 665
q8 9299 1472 1315 1315
q9 4774 4740 4677 4677
q10 6811 1874 1662 1662
q11 472 248 235 235
q12 703 558 476 476
q13 17817 4212 3387 3387
q14 230 236 217 217
q15 954 785 804 785
q16 759 721 688 688
q17 746 870 435 435
q18 5884 5248 5155 5155
q19 1125 967 608 608
q20 500 495 390 390
q21 5021 1961 1523 1523
q22 367 317 255 255
Total cold run time: 96915 ms
Total hot run time: 28711 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4597 4600 4565 4565
q2 q3 1841 2220 1764 1764
q4 902 1165 782 782
q5 4027 4416 4384 4384
q6 185 178 160 160
q7 1816 1645 1552 1552
q8 2464 2683 2553 2553
q9 7696 7296 7413 7296
q10 2618 2805 2411 2411
q11 515 437 409 409
q12 547 736 458 458
q13 3928 4411 3574 3574
q14 282 293 279 279
q15 879 807 832 807
q16 733 776 735 735
q17 1209 1524 1277 1277
q18 7226 6813 6473 6473
q19 884 843 861 843
q20 2089 2154 2041 2041
q21 4019 3541 3433 3433
q22 478 454 448 448
Total cold run time: 48935 ms
Total hot run time: 46244 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org