You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/31 09:41:00 UTC
[GitHub] [incubator-doris] carlvinhust2012 opened a new pull request, #9887: [Enhancement] add some metrics for cpu and memory
carlvinhust2012 opened a new pull request, #9887:
URL: https://github.com/apache/incubator-doris/pull/9887
# Proposed changes
1. add some metrics for cpu monitor;
2. add metrics for process state monitor;
3. add metrics for memory monitor;
It is convenient for us to use grafana to filter through different conditions.
After the added, we can find the cpu metrics like this：
# TYPE doris_be_cpu counter
doris_be_cpu{device="cpu1",mode="guest_nice"} 0
doris_be_cpu{device="cpu1",mode="guest"} 0
doris_be_cpu{device="cpu1",mode="steal"} 0
doris_be_cpu{device="cpu1",mode="soft_irq"} 107168
doris_be_cpu{device="cpu1",mode="irq"} 0
doris_be_cpu{device="cpu1",mode="iowait"} 3726931
doris_be_cpu{device="cpu1",mode="idle"} 2358039214
doris_be_cpu{device="cpu1",mode="system"} 58699464
doris_be_cpu{device="cpu1",mode="nice"} 1700438
doris_be_cpu{device="cpu1",mode="user"} 54974091
we can find the memory metrics like this：
# TYPE doris_be_memory_swpcache gauge
doris_be_memory_swpcache 0
# TYPE doris_be_memory_pswpin gauge
doris_be_memory_pswpin 167785
# TYPE doris_be_memory_pswpout gauge
doris_be_memory_pswpout 203724
# TYPE doris_be_memory_pgpgout gauge
doris_be_memory_pgpgout 152101956232
# TYPE doris_be_memory_cache gauge
doris_be_memory_cache 86546268
# TYPE doris_be_memory_memfree gauge
doris_be_memory_memfree 23520128
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (No)
5. Has unit tests been added: (No)
6. Has document been added or modified: (No)
7. Does it need to update dependencies: (No)
8. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r887522946
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
> after update, just use /proc/vmstat to get system memory.
Why use system memory stats as doris metrics. This should use VM monitoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#issuecomment-1150614681
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r885603919
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
Review Comment:
because I want to keep the style with the old code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r886271652
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
Doris currently uses TCMalloc. Using the following cmd, can get the memory counted by TCMalloc (`http://be_ip:be_httpport/memz`), but this does not include mmap
`MallocExtension::instance()->GetStats`
`MallocExtension::instance()->GetNumericProperty`
Memory statistics including mmap: `http://be_ip:be_httpport/mem_tracker`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r885588770
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
Review Comment:
Why don't we use C++ style file read api?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r887519286
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
after update, just use /proc/vmstat to get system memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
caiconghui commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r887532453
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
for system metric, I think it is better to use thirdparty tool to do system metric collection instead of integrated into doris, it is a common work for all system manager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
morningman merged PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#issuecomment-1150614665
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r887530476
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
just like the cpu metrics, it is also not just belong to doris. so I think the memory stats in /proc/vmstat could be capture d here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r887534503
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
> for system metric, I think it is better to use thirdparty tool to do system metric collection instead of integrated into doris, it is a common work for all system manager
+1, CPU metrics are the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r885611222
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
`/proc/meminfo` is system memory statistics, which should not be used as metrics for doris BE process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r886274798
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
> Doris currently uses TCMalloc. Using the following cmd, can get the memory counted by TCMalloc (`http://be_ip:be_httpport/memz`), but this does not include mmap `MallocExtension::instance()->GetStats` `MallocExtension::instance()->GetNumericProperty`
>
> Memory statistics including mmap: `http://be_ip:be_httpport/mem_tracker`
you means that we no need add the statistics in /proc/meminfo, and just use "http://be_ip:be_httpport/mem_tracker" is enough, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r887499108
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
> you means that we no need add the statistics in /proc/meminfo, and just use "http://be_ip:be_httpport/mem_tracker" is enough, right?
no,
In this pr, use `/proc/vmstat` and `/proc/meminfo` to get system memory or process memory?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] carlvinhust2012 commented on a diff in pull request #9887: [Enhancement] add some metrics for cpu and memory
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #9887:
URL: https://github.com/apache/incubator-doris/pull/9887#discussion_r887530476
##########
be/src/util/system_metrics.cpp:
##########
@@ -718,4 +788,174 @@ void SystemMetrics::get_max_net_traffic(const std::map<std::string, int64_t>& ls
*send_rate = max_send / interval_sec;
*rcv_rate = max_rcv / interval_sec;
}
+
+void SystemMetrics::_install_proc_metrics(MetricEntity* entity) {
+ _proc_metrics.reset(new ProcMetrics(entity));
+}
+
+void SystemMetrics::_update_proc_metrics() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_stat_path, "r");
+#else
+ FILE* fp = fopen("/proc/stat", "r");
+#endif
+ if (fp == nullptr) {
+ char buf[64];
+ LOG(WARNING) << "open /proc/stat failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ return;
+ }
+
+ uint64_t inter = 0, ctxt = 0, procs_r = 0, procs_b = 0;
+ while (getline(&_line_ptr, &_line_buf_size, fp) > 0) {
+ char* start_pos = nullptr;
+ start_pos = strstr(_line_ptr, "intr ");
+ if (start_pos) {
+ sscanf(start_pos, "intr %lu", &inter);
+ _proc_metrics->proc_interrupt->set_value(inter);
+ }
+
+ start_pos = strstr(_line_ptr, "ctxt ");
+ if (start_pos) {
+ sscanf(start_pos, "ctxt %lu", &ctxt);
+ _proc_metrics->proc_ctxt_switch->set_value(ctxt);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_running ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_running %lu", &procs_r);
+ _proc_metrics->proc_procs_running->set_value(procs_r);
+ }
+
+ start_pos = strstr(_line_ptr, "procs_blocked ");
+ if (start_pos) {
+ sscanf(start_pos, "procs_blocked %lu", &procs_b);
+ _proc_metrics->proc_procs_blocked->set_value(procs_b);
+ }
+ }
+
+ if (ferror(fp) != 0) {
+ char buf[64];
+ LOG(WARNING) << "getline failed, errno=" << errno
+ << ", message=" << strerror_r(errno, buf, 64);
+ }
+
+ fclose(fp);
+}
+
+void SystemMetrics::get_metrics_from_proc_meminfo() {
+#ifdef BE_TEST
+ FILE* fp = fopen(k_ut_meminfo_path, "r");
+#else
+ FILE* fp = fopen("/proc/meminfo", "r");
Review Comment:
just like the cpu metrics, it is also not just belong to doris. so I think the memory stats in /proc/vmstat could be captured here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org