You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/02 04:31:54 UTC
[GitHub] [incubator-doris] acelyc111 opened a new pull request #3246:
[metrics] Add some metrics for container size in BE
acelyc111 opened a new pull request #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246
We can observe the workload of BE, and also it's a way to check
whether there is any problem in BE, like some container increase
to large and lead to OOM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on issue #3246: [metrics] Add
some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-612606111
> > > I have a question about dynamic collect the metric.
> > > If people can remember so many metrics and use http server to get it?
> > > Why not add it into DorisMetric instead?
> >
> >
> > Yes, I register all of these metrics to DorisMetric, via a new added macro in be/src/util/doris_metrics.h., and user can visit '/metrics' to obtain all metrics.
>
> I mean you can write the variable in doris_metrics.h, it's can be a unified entrance.
> The followers can easy know metrics in doris through reading doris_metris.h.
Good advice, done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #3246:
[metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-612625122
> OK! And I want to know that is there a doc describing all the exsited metrics? If not, I think we could provide a way, for example CLI, to dump all metrics, with their description which has been well written when the related metric is defined/registered, then we only need to maintain the code, without an independent doc.
It's OK to not add doc in this PR. Currently we do not have a doc for these metrics. We can add this doc in another PR later.
But it's better to list the new metrics you added in this PR in your commit log and make a brief description of them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #3246: [metrics]
Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-611934466
@acelyc111 Hi, Would better add a doc to list all new metrics and describe the metrics meaning. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3246: [metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#discussion_r402117572
##########
File path: be/src/runtime/result_buffer_mgr.cpp
##########
@@ -33,6 +34,11 @@ namespace doris {
ResultBufferMgr::ResultBufferMgr()
: _is_stop(false) {
+ REGISTER_PRIVATE_VARIABLE_METRIC(result_buffer_block_count);
+ DorisMetrics::metrics()->register_hook("result_buffer_block_count", [&]() {
+ boost::lock_guard<boost::mutex> l(_lock);
Review comment:
```suggestion
std::lock_guard<boost::mutex> l(_lock);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 edited a comment on issue #3246:
[metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 edited a comment on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-610160975
> I have a question about dynamic collect the metric.
> If people can remember so many metrics and use http server to get it?
> Why not add it into DorisMetric instead?
Yes, I register all of these metrics to DorisMetric, via a new added macro in be/src/util/doris_metrics.h., and user can visit '/metrics' to obtain all metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on issue #3246: [metrics] Add
some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-614489201
> > OK! And I want to know that is there a doc describing all the exsited metrics? If not, I think we could provide a way, for example CLI, to dump all metrics, with their description which has been well written when the related metric is defined/registered, then we only need to maintain the code, without an independent doc.
>
> It's OK to not add doc in this PR. Currently we do not have a doc for these metrics. We can add this doc in another PR later.
>
> But it's better to list the new metrics you added in this PR in your commit log and make a brief description of them.
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request
#3246: [metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#discussion_r402116147
##########
File path: be/src/runtime/etl_job_mgr.cpp
##########
@@ -62,6 +63,21 @@ const std::string ERROR_FILE_PREFIX = "error_log";
EtlJobMgr::EtlJobMgr(ExecEnv* exec_env) :
_exec_env(exec_env), _success_jobs(5000), _failed_jobs(5000) {
+ REGISTER_PRIVATE_VARIABLE_METRIC(running_etl_job_count);
Review comment:
`EtlJobMgr` is deprecated, it will be removed in next version.
So you don't need to add metrics for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on issue #3246: [metrics] Add
some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-610160975
> I have a question about dynamic collect the metric.
> If people can remember so many metrics and use http server to get it?
> Why not add it into DorisMetric instead?
Yes, I register all of these metrics to DorisMetric, via a new added macro in be/src/util/doris_metrics.h., and user and access '/metrics' to obtain all metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on issue #3246: [metrics] Add
some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-612574756
> @acelyc111 Hi, Would better add a doc to list all new metrics and describe the metrics meaning. Thanks.
OK! And I want to know that is there a doc describing all the exsited metrics? If not, I think we could provide a way, for example CLI, to dump all metrics, with their description which has been well written when the related metric is defined/registered, then we only need to maintain the code, without an independent doc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3246: [metrics] Add
some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-609742521
I have a question about dynamic collect the metric.
If people can remember so many metrics and use http server to get it?
Why not add it into DorisMetric instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 edited a comment on issue #3246:
[metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 edited a comment on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-612606111
> > > I have a question about dynamic collect the metric.
> > > If people can remember so many metrics and use http server to get it?
> > > Why not add it into DorisMetric instead?
> >
> >
> > Yes, I register all of these metrics to DorisMetric, via a new added macro in be/src/util/doris_metrics.h., and user can visit '/metrics' to obtain all metrics.
>
> I mean you can write the variable in doris_metrics.h, it's can be a unified entrance.
> The followers can easy know metrics in doris through reading doris_metris.h.
@chaoyli Good advice, done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3246: [metrics]
Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-612625122
> OK! And I want to know that is there a doc describing all the exsited metrics? If not, I think we could provide a way, for example CLI, to dump all metrics, with their description which has been well written when the related metric is defined/registered, then we only need to maintain the code, without an independent doc.
It's OK to not add doc in this PR. Currently we do not have a doc for these metrics. We can add this doc in another PR later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on issue #3246: [metrics] Add
some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-611588452
> LGTM
> @acelyc111, Hi could you please resolve the conflicts?
Resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 edited a comment on issue #3246:
[metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 edited a comment on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-612574756
> @acelyc111 Hi, Would better add a doc to list all new metrics and describe the metrics meaning. Thanks.
@kangkaisen
OK! And I want to know that is there a doc describing all the exsited metrics? If not, I think we could provide a way, for example CLI, to dump all metrics, with their description which has been well written when the related metric is defined/registered, then we only need to maintain the code, without an independent doc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3246: [metrics]
Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-611563419
LGTM
@acelyc111, Hi could you please resolve the conflicts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3246: [metrics] Add
some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3246: [metrics] Add some metrics for container size in BE
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-612282382
> > I have a question about dynamic collect the metric.
> > If people can remember so many metrics and use http server to get it?
> > Why not add it into DorisMetric instead?
>
> Yes, I register all of these metrics to DorisMetric, via a new added macro in be/src/util/doris_metrics.h., and user can visit '/metrics' to obtain all metrics.
I mean you can write the variable in doris_metrics.h, it's can be a unified entrance.
The followers can easy know metrics in doris through reading doris_metris.h.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on issue #3246: [metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on issue #3246:
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-617880130
> hi @acelyc111 Please resolve the conflicts
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3246: [metrics] Add some metrics for container size in BE
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3246:
URL: https://github.com/apache/incubator-doris/pull/3246#issuecomment-617833512
hi @acelyc111 Please resolve the conflicts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org