You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/02/15 13:49:51 UTC
[GitHub] [skywalking] JayeshMuley opened a new issue #6370: Alarm compression rules not working
JayeshMuley opened a new issue #6370:
URL: https://github.com/apache/skywalking/issues/6370
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [X ] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
Have setup the composite rule but alarms are not coming although Time-Bucket is overlapping , Please help if anything wrong in configuration. I am using Skywaling8.3
Configuration -
alarm-settings.yml
rules:
service_instance_resp_time_alert_rule:
metrics-name: service_instance_resp_time
op: ">"
threshold: 100
period: 5
count: 1
message: Response time of is more than 100ms in 1 count of last 5 minutes Message1
service_instance_jvm_cpu__alert_rule:
metrics-name: instance_jvm_cpu
op: ">"
threshold: 20
period: 5
count: 1
message: service_instance_jvm_cpu is more than 20% in 1 times in last 5 minutes Message2
composite-rules:
comp_rule:
expression: service_instance_resp_time_alert_rule && service_instance_jvm_cpu__alert_rule
message: Message4
expression: (service_instance_resp_time_alert_rule || service_instance_jvm_cpu__alert_rule)
message: Message5
expression: service_instance_resp_time_alert_rule
message: Message7
expression: service_instance_jvm_cpu__alert_rule
message: Message8
expression: service_instance_jvm_cpu__alert_rule && service_instance_jvm_memory_heap_alert_rule
message: Message6
individual alarm in database -
CPU Alert
+---------------+----------------+--------------------------------------------------------------------------+
| start_time | time_bucket | substr(alarm_message,10,1000) |
+---------------+----------------+--------------------------------------------------------------------------+
| 1613134640445 | 20210212182720 | nstance_jvm_cpu is more than 20% in 1 times in last 5 minutes Message2 |
| 1613135000445 | 20210212183320 | nstance_jvm_cpu is more than 20% in 1 times in last 5 minutes Message2 |
| 1613135360445 | 20210212183920 | nstance_jvm_cpu is more than 20% in 1 times in last 5 minutes Message2 |
| 1613135720445 | 20210212184520 | nstance_jvm_cpu is more than 20% in 1 times in last 5 minutes Message2 |
| 1613136620445 | 20210212190020 | nstance_jvm_cpu is more than 20% in 1 times in last 5 minutes Message2 |
+---------------+----------------+--------------------------------------------------------------------------+
Responce Time Alert
+---------------+----------------+---------------------------------------------------------------------+
| start_time | time_bucket | substr(alarm_message,10,1000) |
+---------------+----------------+---------------------------------------------------------------------+
| 1613134460446 | 20210212182420 | time of is more than 100ms in 1 times of last 5 minutes Message1 |
| 1613134820445 | 20210212183020 | time of is more than 100ms in 1 times of last 5 minutes Message1 |
| 1613135540445 | 20210212184220 | time of is more than 100ms in 1 times of last 5 minutes Message1 |
+---------------+----------------+---------------------------------------------------------------------+
3 rows in set (0.05 sec)
But no Composite alerts
Composite Alert for Message4,5,6,7,8 are not detected.
Thanks and Regards,
Jayesh
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] JayeshMuley edited a comment on issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
JayeshMuley edited a comment on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-782829236
sorry for confusion . Below is the Response Time and CPU data and
Response Time -
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message1%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893462668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221131422 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893822668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132022 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894182668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132622 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.02 sec)
CPU Util -
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message3%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893522668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221131522 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893882668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221132122 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894242668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221132722 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.06 sec)
But no Composite data
mysql> select * from alarm_record where alarm_message like '%Message4%';
Empty set (0.00 sec)
Thanks and Regards,
Jayesh
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6370: Alarm compression rules not working
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-779240455
@xbkaishui Anything you can see from the data?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #6370: Alarm compression rules not working
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-779520736
> ```yaml
> composite-rules:
> comp_rule:
> expression: service_instance_resp_time_alert_rule && service_instance_jvm_cpu__alert_rule
> message: Message4
> expression: (service_instance_resp_time_alert_rule || service_instance_jvm_cpu__alert_rule)
> message: Message5
> expression: service_instance_resp_time_alert_rule
> message: Message7
> expression: service_instance_jvm_cpu__alert_rule
> message: Message8
> expression: service_instance_jvm_cpu__alert_rule && service_instance_jvm_memory_heap_alert_rule
> message: Message6
> ```
This is not a valid yaml file (b/c you have duplicate keys `expression` and `message` under section `comp_rule`), try this
```yaml
# ... the other configurations
composite-rules:
comp1_rule:
expression: service_instance_resp_time_alert_rule && service_instance_jvm_cpu__alert_rule
message: Message4
comp2_rule:
expression: (service_instance_resp_time_alert_rule || service_instance_jvm_cpu__alert_rule)
message: Message5
comp3_rule:
expression: service_instance_resp_time_alert_rule
message: Message7
comp4_rule:
expression: service_instance_jvm_cpu__alert_rule
message: Message8
comp5_rule:
expression: service_instance_jvm_cpu__alert_rule && service_instance_jvm_memory_heap_alert_rule
message: Message6
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #6370:
URL: https://github.com/apache/skywalking/issues/6370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] JayeshMuley commented on issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
JayeshMuley commented on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-782816941
I still facing this issue where composit alerts are not coming. See the configuration file and Alerts
rules:
service_instance_resp_time_rule:
metrics-name: service_instance_resp_time
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: High Response service Instance {name} Message1
service_instance_cpm_rule:
metrics-name: service_instance_cpm
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: High cpm service Instance Instance {name} Message2
instance_jvm_cpu_rule:
metrics-name: instance_jvm_cpu
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: CPU Util Instance {name} Message3
composite-rules:
comp1_rule:
expression: service_instance_resp_time_rule && instance_jvm_cpu_rule
message: Message4 service Instance {name}
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message1%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893462668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221131422 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893822668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132022 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894182668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132622 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.02 sec)
mysql>
mysql>
mysql>
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message2%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893402669 | High cpm service Instance Instance "DIEPAIOPSITCE | 20210221131322 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893762668 | High cpm service Instance Instance "DIEPAIOPSITCE | 20210221131922 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894122668 | High cpm service Instance Instance "DIEPAIOPSITCE | 20210221132522 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.00 sec)
mysql>
mysql>
mysql> select * from alarm_record where alarm_message like '%Message4%';
Empty set (0.00 sec)
Kindly suggest.
Thanks and Regards,
Jayesh Muley
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] JayeshMuley commented on issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
JayeshMuley commented on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-782829236
sorry for confusion . Below is the Response Time and CPU data and
Response Time -
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message1%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893462668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221131422 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893822668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132022 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894182668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132622 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.02 sec)
CPU Util -
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message3%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893522668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221131522 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893882668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221132122 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894242668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221132722 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.06 sec)
But no Composite data
mysql> select * from alarm_record where alarm_message like '%Message4%';
Empty set (0.00 sec)
Configuration -
rules:
service_instance_resp_time_rule:
metrics-name: service_instance_resp_time
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: High Response service Instance {name} Message1
service_instance_cpm_rule:
metrics-name: service_instance_cpm
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: High cpm service Instance Instance {name} Message2
instance_jvm_cpu_rule:
metrics-name: instance_jvm_cpu
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: CPU Util Instance {name} Message3
composite-rules:
comp1_rule:
expression: service_instance_resp_time_rule && instance_jvm_cpu_rule
message: Message4 service Instance {name}
Thanks and Regards,
Jayesh
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] JayeshMuley edited a comment on issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
JayeshMuley edited a comment on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-782829236
sorry for confusion . Below is the Response Time and CPU data and
Response Time -
```
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message1%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893462668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221131422 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893822668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132022 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894182668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132622 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.02 sec)
```
CPU Util -
```
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message3%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893522668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221131522 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893882668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221132122 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894242668 | CPU Util Instance "DIEPAIOPSITCENT7534" of core_p | 20210221132722 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.06 sec)
```
But no Composite data
mysql> select * from alarm_record where alarm_message like '%Message4%';
Empty set (0.00 sec)
Thanks and Regards,
Jayesh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] JayeshMuley edited a comment on issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
JayeshMuley edited a comment on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-782816941
I still facing this issue where composit alerts are not coming. See the configuration file and Alerts
```yaml
rules:
service_instance_resp_time_rule:
metrics-name: service_instance_resp_time
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: High Response service Instance {name} Message1
service_instance_cpm_rule:
metrics-name: service_instance_cpm
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: High cpm service Instance Instance {name} Message2
instance_jvm_cpu_rule:
metrics-name: instance_jvm_cpu
op: ">"
threshold: 10
period: 5
count: 1
only-as-condition: false
message: CPU Util Instance {name} Message3
composite-rules:
comp1_rule:
expression: service_instance_resp_time_rule && instance_jvm_cpu_rule
message: Message4 service Instance {name}
```
```text
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message1%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893462668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221131422 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893822668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132022 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894182668 | High Response service Instance "DIEPAIOPSITCENT753 | 20210221132622 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.02 sec)
mysql>
mysql>
mysql>
mysql> select name , start_time , substr(alarm_message,1,50) , time_bucket from alarm_record where name like '%core_perf.service%' and alarm_message like '%Message2%';
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| name | start_time | substr(alarm_message,1,50) | time_bucket |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893402669 | High cpm service Instance Instance "DIEPAIOPSITCE | 20210221131322 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613893762668 | High cpm service Instance Instance "DIEPAIOPSITCE | 20210221131922 |
| "DIEPAIOPSITCENT7534" of core_perf.service | 1613894122668 | High cpm service Instance Instance "DIEPAIOPSITCE | 20210221132522 |
+--------------------------------------------+---------------+----------------------------------------------------+----------------+
3 rows in set (0.00 sec)
mysql>
mysql>
mysql> select * from alarm_record where alarm_message like '%Message4%';
Empty set (0.00 sec)
```
Kindly suggest.
Thanks and Regards,
Jayesh Muley
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #6370:
URL: https://github.com/apache/skywalking/issues/6370#issuecomment-782820863
@JayeshMuley you pasted database records of the alarm messages `Message1` and `Message2`, but you configured `service_instance_resp_time_rule && instance_jvm_cpu_rule`, which should be `Message2` and `Message3`, please recheck.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #6370: Alarm composite rules not working
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #6370:
URL: https://github.com/apache/skywalking/issues/6370
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org