You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2021/10/18 11:02:13 UTC
[GitHub] [incubator-pegasus] padmejin opened a new pull request #828: fix coredump when table name contains '_' and prometheus is enabled
padmejin opened a new pull request #828:
URL: https://github.com/apache/incubator-pegasus/pull/828
### What problem does this PR solve? <!--add issue link with summary if exists-->
After we enable prometheus(set "perf_counter_sink" to "prometheus"), pegasus ended up core dumped after restart.
We debuged the core, found out it was cored while analysing table name that contains "_".
This PR fixes the bug by changing the order of formating table names.
### What is changed and how does it work?
Prometheus does not support some special charactor in metric name.
This original code first formated the table name by replacing special charactor to "_" or "@".
Then, it try to parse gpid as prometheus labels by spliting the foramted name by ":" or "_".
Unfortunately, tables that already contains "_" may results in unexpected group result, which eventually ended up core.
### Checklist <!--REMOVE the items that are not applicable-->
##### Tests <!-- At least one of them must be included. -->
- Unit test
- Integration test
- Manual test (add detailed scripts or steps below)
- No code
##### Code changes
- Has exported function/method change
- Has exported variable/fields change
- Has interface methods change
- Has persistent data change
##### Side effects
- Possible performance regression
- Increased code complexity
- Breaking backward compatibility
##### Related changes
- Need to cherry-pick to the release branch
- Need to update the documentation
- Need to be included in the release note
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] padmejin commented on pull request #828: fix: coredump when table name contains '_' and prometheus is enabled
Posted by GitBox <gi...@apache.org>.
padmejin commented on pull request #828:
URL: https://github.com/apache/incubator-pegasus/pull/828#issuecomment-946292504
As this picture shows below, when core dumps it was trying to analyze metric naming `replica_econ_replica_table_level_RPC_RRDB_RRDB_REMOVE_latency_ns_:impala_historical_profile_p999`, where `impala_historical_profile` is a table we created earlier that contains multiple underscore.
As the code shown below on line 256, it tries to split metrics_name by `:` first, then split the last part by `_` and put the splited tokens in an array that length is 3. Because this table already contains underscore, there are more than 3 tokens, so it got core dumped after at line 259 trying to set the fourth element of the array.
This patch fix it by switching these format order:
Firstly, it try to parse gpid as prometheus labels by spliting the foramted name by `:` or `@`.
Then, it formated the table name by replacing special charactor to `_` or "@".
In this order tables that contains `_` can be parsed expectedly in the first step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 merged pull request #828: fix: coredump when table name contains '_' and prometheus is enabled
Posted by GitBox <gi...@apache.org>.
acelyc111 merged pull request #828:
URL: https://github.com/apache/incubator-pegasus/pull/828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on a change in pull request #828: fix: coredump when table name contains '_' and prometheus is enabled
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on a change in pull request #828:
URL: https://github.com/apache/incubator-pegasus/pull/828#discussion_r731460375
##########
File path: src/reporter/pegasus_counter_reporter.cpp
##########
@@ -268,6 +264,13 @@ void pegasus_counter_reporter::update()
// create metrics that prometheus support to report data
metrics_name = lv.front() + app[2];
+
+ // prometheus metric_name don't support characters like .*()@, it only support ":"
+ // and "_"
+ // so change the name to make it all right
+ format_metrics_name(metrics_name);
+
+
Review comment:
```suggestion
```
##########
File path: src/reporter/pegasus_counter_reporter.cpp
##########
@@ -234,22 +234,18 @@ void pegasus_counter_reporter::update()
const dsn::perf_counters::counter_snapshot &cs) {
std::string metrics_name = cs.name;
- // prometheus metric_name don't support characters like .*()@, it only support ":"
- // and "_"
- // so change the name to make it all right
- format_metrics_name(metrics_name);
-
- // split metric_name like "collector_app_pegasus_app_stat_multi_put_qps:1_0_p999" or
- // "collector_app_pegasus_app_stat_multi_put_qps:1_0"
+ // split metric_name like "collector*app.pegasus*app_stat_multi_put_qps@1.0.p999" or
+ // "collector*app.pegasus*app_stat_multi_put_qps@1.0"
Review comment:
```suggestion
// Splits metric_name like:
// "collector*app.pegasus*app_stat_multi_put_qps@1.0.p999"
// "collector*app.pegasus*app_stat_multi_put_qps@1.0"
```
##########
File path: src/reporter/pegasus_counter_reporter.cpp
##########
@@ -268,6 +264,13 @@ void pegasus_counter_reporter::update()
// create metrics that prometheus support to report data
metrics_name = lv.front() + app[2];
+
+ // prometheus metric_name don't support characters like .*()@, it only support ":"
Review comment:
```suggestion
// prometheus metric_name doesn't support characters like .*()@, it only supports ":"
```
##########
File path: src/reporter/pegasus_counter_reporter.cpp
##########
@@ -268,6 +264,13 @@ void pegasus_counter_reporter::update()
// create metrics that prometheus support to report data
metrics_name = lv.front() + app[2];
+
+ // prometheus metric_name don't support characters like .*()@, it only support ":"
+ // and "_"
+ // so change the name to make it all right
Review comment:
```suggestion
// and "_" so change the name to make it all right.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org