You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/01/13 07:15:07 UTC
[GitHub] [dolphinscheduler] liqingwang opened a new issue #7992: [Feature][Alter] Support PagerDuty Plugin
liqingwang opened a new issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
Hi, All.
We want to add an alert plugin named `PagerDuty` for DolphinScheduler. And we would like to contribute to the community.
There is one question that needs to be discussed.
There is two alert strategies `success` & `fail` for process or task. One record will be inserted in t_ds_alert when triggered. And the record doesn't contain the alert strategy. Other alert plugins don't care about alert strategy, they all send the title and content to where they are configured.
And there is a param named `severity` in PagerDuty API, which in the official documentation means :
> The perceived severity of the status the event is describing with respect to the affected system. This can be `critical`, `error`, `warning` or `info`.
We think that when somebody configured the alert strategy of process or task is `success`, it means when the process or task succeeded, they just want a massage, this situation should use `severity: info` calls the PagerDuty API.
When they configured the strategy as `fail`, calling the PagerDuty Api with `critical` or `error` is better.
But if support this case, we need to change the code. Add a column in the t_ds_alert table and change the logic insert the strategy field in this table when the alert strategy is triggered.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lenboo edited a comment on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
lenboo edited a comment on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1014266303
@liqingwang Good job!
_Add alert_strategy in t_ds_alert table_
According to my understanding, this field should be "process/task state"?
When process/task finish, alert can only be triggered once, and 't_ds_alert' will only have one data about this process/task.
> 3. Change the code to bind the alert alarm instance strategy to process strategy
> Bind alarm instance's strategy to t_ds_process_instance's warning_type and t_ds_schedules's > warning_type.
> When a process with a configured notification strategy triggers an alert. A new record will insert > in t_ds_alert table saving the triggered strategy, whether the process is successful or failure. The > configured warning group will be triggered. The alarm instance will be triggered one by one.
> And there is a new logic. Only process's strategy is matched alarm instance's strategy. The alert > will be sent, otherwise, the alert will be ignored._
The alert instances can be sent according to state and strategy.
At the same time, we need to consider not only success and failure, but also others, such as pause and stop, and possible increase in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lenboo commented on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
lenboo commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1014266303
@liqingwang Good job!
_Add alert_strategy in t_ds_alert table_
According to my understanding, this field should be "process/task state"?
When process/task finish, alert can only be triggered once, and 't_ds_alert' will only have one data about this process/task.
_3. Change the code to bind the alert alarm instance strategy to process strategy
Bind alarm instance's strategy to t_ds_process_instance's warning_type and t_ds_schedules's warning_type.
When a process with a configured notification strategy triggers an alert. A new record will insert in t_ds_alert table saving the triggered strategy, whether the process is successful or failure. The configured warning group will be triggered. The alarm instance will be triggered one by one.
And there is a new logic. Only process's strategy is matched alarm instance's strategy. The alert will be sent, otherwise, the alert will be ignored._
The alert instances can be sent according to state and strategy.
At the same time, we need to consider not only success and failure, but also others, such as pause and stop, and possible increase in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #7992: [Feature][Alter] Support PagerDuty Plugin
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1014187177
There is a situation in our scenario. Defined a task and when the task succeeds, send a message to the WeChat group, and call the owner via PagerDuty when the task fails. Call different notification plug-ins with different task statuses.
DolphinScheduler is very suitable for this kind of scenario. It relies on saving the alert strategy in t_ds_alert when triggered like above.
### 1. Here is the design:
#### 1. Add alert_strategy in t_ds_alert table
alert_strategy tinyint(4) NOT NULL COMMENT 'Alert strategy: 0 process is successfully, 1 process is failed'.
Save the strategy when the alert is triggered.
#### 2. Add alert_strategy in t_ds_alert_plugin_instance table
alert_strategy tinyint(4) NOT NULL COMMENT 'Alarm strategy: 1 process is sent successfully, 2 process is sent failed, 3 process is sent successfully and all failures are sent'.
Add strategy choice when creating alarm instance. The default value is 3 processes is sent successfully and all failures are sent. For compatibility with older versions.
#### 3. Change the code to bind the alert alarm instance strategy to process strategy
Bind alarm instance's strategy to t_ds_process_instance's warning_type and t_ds_schedules's warning_type.
When a process with a configured notification strategy triggers an alert. A new record will insert in t_ds_alert table saving the triggered strategy, whether the process is successful or failure. The configured warning group will be triggered. The alarm instance will be triggered one by one.
And there is a new logic. Only process's strategy is matched alarm instance's strategy. The alert will be sent, otherwise, the alert will be ignored.
#### 4. For example
When the process's warning_type is 1 process is sent successfully.
And the alarm group has three alarm instances **a & b & c**. The **a's** alert_strategy is **1 process is sent successfully**. The **b's** alert_strategy is **2 process is sent failed**. The **c's** alert_strategy is **3 process is sent successfully and all failures are sent**.
- When the process is **successful**, the alarm instance **a and c** will **send**, the alarm instance **b** will **ignore**.
- When the process **fails**, the alarm instances **b and c** will **send**, the alarm instance **a** will **ignore**.
### 2. Here is the plan:
We want to part this solution into three-part.
1. Add PargerDuty plugin and call API with severity: critical as above
2. Add alert_strategy column in t_ds_alert table & Save the alert strategy when triggered
3. Add alert_strategy column in t_ds_alert_plugin_instance table & Change the login to match strategy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang edited a comment on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
liqingwang edited a comment on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1015136643
@lenboo Thanks for your reply!
> Add alert_strategy in t_ds_alert table
> According to my understanding, this field should be "process/task state"?
Yes, you are right. Maybe change the COMMENT to below is more clear.
`alert_strategy tinyint(4) NOT NULL COMMENT 'Alert strategy: 0 process/task is successfully, 1 process/task is failed'.`
> When process/task finish, alert can only be triggered once, and 't_ds_alert' will only have one data about this process/task.
That's right. Only one record will be inserted in 't_ds_alert' when process/task is finished. But process/task binds the alert group, and multiple instances of multiple alert methods can be created in an alert group. The design document is [here](https://github.com/apache/dolphinscheduler/issues/3049#issue-644380894). The source code is [here](https://github.com/apache/dolphinscheduler/blob/2.0.2-release/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSender.java#L54).
> At the same time, we need to consider not only success and failure, but also others, such as pause and stop, and possible increase in the future
Yes, I agree with you. And I think we can do step-by-step iterations. In the further, we can support more scenes, such as pause and stop you said, and maybe online and offline. The current architecture is supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1015136643
@lenboo Thanks for your replay!
> Add alert_strategy in t_ds_alert table
> According to my understanding, this field should be "process/task state"?
Yes, you are right. Maybe change the COMMENT to below is more clear.
`alert_strategy tinyint(4) NOT NULL COMMENT 'Alert strategy: 0 process/task is successfully, 1 process/task is failed'.`
> When process/task finish, alert can only be triggered once, and 't_ds_alert' will only have one data about this process/task.
That's right. Only one record will be inserted in 't_ds_alert' when process/task is finished. But process/task binds the alert group, and multiple instances of multiple alert methods can be created in an alert group. The design document is [here](https://github.com/apache/dolphinscheduler/issues/3049#issue-644380894). The source code is [here](https://github.com/apache/dolphinscheduler/blob/2.0.2-release/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSender.java#L54).
> At the same time, we need to consider not only success and failure, but also others, such as pause and stop, and possible increase in the future
Yes, I agree with you. And I think we can step-by-step iterations. In the futher, we can support more scenes, such as pause and stop you said, and maybe online and offline. The current architecture is supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1016230582
The first part is ready to be reviewed. #8120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #7992: [Feature][Alter] Support PagerDuty Plugin
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1011883929
> It's just an alert plugin that has more detailed granularity.
Do we need to specify it when the task is executed? This granularity seems to be the general granularity of alert, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #7992: [Feature][Alter] Support PagerDuty Plugin
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1011860242
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] gaojun2048 closed issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
gaojun2048 closed issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #7992: [Feature][Alter] Support PagerDuty Plugin
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1011872396
Hi, I have a question, is this a plugin? and if so, what is its main function on the plugin side? Or is it just that the granularity of alert services is more detailed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1014212824
> There is a situation in our scenario. Defined a task and when the task succeeds, send a message to the WeChat group, and call the owner via PagerDuty when the task fails. Call different notification plug-ins with different task statuses.
>
> DolphinScheduler is very suitable for this kind of scenario. It relies on saving the alert strategy in t_ds_alert when triggered like above.
>
> ### 1. Here is the design:
> #### 1. Add alert_strategy in t_ds_alert table
> alert_strategy tinyint(4) NOT NULL COMMENT 'Alert strategy: 0 process is successfully, 1 process is failed'.
>
> Save the strategy when the alert is triggered.
>
> #### 2. Add alert_strategy in t_ds_alert_plugin_instance table
> alert_strategy tinyint(4) NOT NULL COMMENT 'Alarm strategy: 1 process is sent successfully, 2 process is sent failed, 3 process is sent successfully and all failures are sent'.
>
> Add strategy choice when creating alarm instance. The default value is 3 processes is sent successfully and all failures are sent. For compatibility with older versions.
>
> #### 3. Change the code to bind the alert alarm instance strategy to process strategy
> Bind alarm instance's strategy to t_ds_process_instance's warning_type and t_ds_schedules's warning_type.
>
> When a process with a configured notification strategy triggers an alert. A new record will insert in t_ds_alert table saving the triggered strategy, whether the process is successful or failure. The configured warning group will be triggered. The alarm instance will be triggered one by one.
>
> And there is a new logic. Only process's strategy is matched alarm instance's strategy. The alert will be sent, otherwise, the alert will be ignored.
>
> #### 4. For example
> When the process's warning_type is 1 process is sent successfully.
>
> And the alarm group has three alarm instances **a & b & c**. The **a's** alert_strategy is **1 process is sent successfully**. The **b's** alert_strategy is **2 process is sent failed**. The **c's** alert_strategy is **3 process is sent successfully and all failures are sent**.
>
> * When the process is **successful**, the alarm instance **a and c** will **send**, the alarm instance **b** will **ignore**.
> * When the process **fails**, the alarm instances **b and c** will **send**, the alarm instance **a** will **ignore**.
>
> ### 2. Here is the plan:
> We want to part this solution into three-part.
>
> 1. Add PargerDuty plugin and call API with severity: critical as above
> 2. Add alert_strategy column in t_ds_alert table & Save the alert strategy when triggered
> 3. Add alert_strategy column in t_ds_alert_plugin_instance table & Change the login to match strategy
+1, Good job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang edited a comment on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
liqingwang edited a comment on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1014187177
There is a situation in our scenario. Defined a task and when the task succeeds, send a message to the WeChat group, and call the owner via PagerDuty when the task fails. Call different notification plug-ins with different task statuses.
DolphinScheduler is very suitable for this kind of scenario. It relies on saving the alert strategy in t_ds_alert when triggered like above.
### 1. Here is the design:
#### 1. Add alert_strategy in t_ds_alert table
alert_strategy tinyint(4) NOT NULL COMMENT 'Alert strategy: 0 process is successfully, 1 process is failed'.
Save the strategy when the alert is triggered.
#### 2. Add alert_strategy in t_ds_alert_plugin_instance table
alert_strategy tinyint(4) NOT NULL COMMENT 'Alarm strategy: 1 process is sent successfully, 2 process is sent failed, 3 process is sent successfully and all failures are sent'.
Add strategy choice when creating alarm instance. The default value is 3 processes is sent successfully and all failures are sent. For compatibility with older versions.
#### 3. Change the code to bind the alert alarm instance strategy to process strategy
Bind alarm instance's strategy to t_ds_process_instance's warning_type and t_ds_schedules's warning_type.
When a process with a configured notification strategy triggers an alert. A new record will insert in t_ds_alert table saving the triggered strategy, whether the process is successful or failure. The configured warning group will be triggered. The alarm instance will be triggered one by one.
And there is a new logic. Only process's strategy is matched alarm instance's strategy. The alert will be sent, otherwise, the alert will be ignored.
#### 4. For example
When the process's warning_type is 1 process is sent successfully.
And the alarm group has three alarm instances **a & b & c**. The **a's** alert_strategy is **1 process is sent successfully**. The **b's** alert_strategy is **2 process is sent failed**. The **c's** alert_strategy is **3 process is sent successfully and all failures are sent**.
- When the process is **successful**, the alarm instance **a and c** will **send**, the alarm instance **b** will **ignore**.
- When the process **fails**, the alarm instances **b and c** will **send**, the alarm instance **a** will **ignore**.
### 2. Here is the plan:
We want to part this solution into three-part.
1. Add PargerDuty plugin and call API with severity: critical as above
2. Add alert_strategy column in t_ds_alert table & Save the alert strategy when triggered
3. Add alert_strategy column in t_ds_alert_plugin_instance table & Change the logic to match strategy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #7992: [Feature][Alert] Support PagerDuty Plugin && Alert module judging strategy
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1055248746
Hi All.
I have finished the code and the design needs to be changed a bit.
#### 1. Add warning_type in t_ds_alert table
```sql
ALTER TABLE `t_ds_alert` ADD COLUMN `warning_type` tinyint(4) DEFAULT '2' COMMENT '1 process is successfully, 2 process/task is failed';
```
Save the warning type when the alert is triggered. And the default value is 2.
The reason why use warning_type because of we have this enum warning_type in the code, and we don't need to create the new enum.
And in the future when we want to add new alert situation we just needs to change one place code.
#### 2. Don't need to add column in t_ds_alert_plugin_instance table
I add a new input param named `warningType` in AlertPluginManager class. And installPlugin function will add this params to all plugins.
Has three choice, success fail all. The default value is all.
The judging strategy is same as above, when the plugin params has no `warningType` field and the default value is all to compatible with historical logic.
And I merged Plan 2 Save the alert strategy when triggered & Plan 3 Change the login to match strategy.
Please to be reviewed. #8636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #7992: [Feature][Alter] Support PagerDuty Plugin
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1011873675
It's just an alert plugin that has more detailed granularity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #7992: [Feature][Alter] Support PagerDuty Plugin
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #7992:
URL: https://github.com/apache/dolphinscheduler/issues/7992#issuecomment-1011916557
DolphinScheduler has specified the refined notification strategy when the task is executed.
We want to bind DolphinScheduler to PagerDuty's granularity.
DolphinScheduler's `success `-> PagerDuty's `info`
DolphinScheduler's `fail` -> PagerDuty's `critical or error`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org