You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/05/19 06:41:30 UTC
[GitHub] [dolphinscheduler] wangxj3 opened a new issue #5525: server down will send repetitive message
wangxj3 opened a new issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525
The master and worker will listen to other master or worker services down from zk, and will send message notifications, but this message will send one for each monitored service, so repeat.
For example, I have 3 masters and 3 workers, and one of the masters is down. The remaining five services will insert an alert data
in the database.
I haven't a good plan to fix this issue ,shoule we consider the influence of normal version iterations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525#issuecomment-843792655
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525#issuecomment-843833139
@wangxj3 Yeal, I also noticed this problem, this caused by each instance will listen the node path in zookeeper.
One possible solution is to get the lock on zookeeper when insert the instance crash alert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525#issuecomment-844025537
@wangxj3 I think it is hard to get the exact time when the server down, the time that each instance gets this event may be different.
Use lock on zookeeper does not seem to be a good solution for this issue, because some instances may get this event late.
The best solution may be when we insert to the t_ds_alert table, check whether there is the same content in the database for sending, the SQL like below
```sql
INSERT INTO t_ds_alert(instance, user, item)
SELECT xx, xxx, xxxx FROM t_ds_alert
WHERE NOT EXISTS (SELECT * FROM t_ds_alert WHERE content ='' AND alert_status = WAIT_EXECUTION)
```
I didn't test this SQL, but this should be work.
In this way, there will be only one unsent alert for each instance in the database. But I think this is reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun edited a comment on issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
ruanwenjun edited a comment on issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525#issuecomment-843833139
@wangxj3 Yeal, I also noticed this problem, this caused by each instance will listen the node path in zookeeper.
~~One possible solution is to get the lock on zookeeper when insert the instance crash alert.~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun edited a comment on issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
ruanwenjun edited a comment on issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525#issuecomment-843833139
@wangxj3 Yeal, I also noticed this problem, this caused by each instance will listen the node path in zookeeper.
~~One possible solution is to get the lock on zookeeper when insert the instance crash alert.~~
Can we think each instance should save at most one unsent instance alert in database?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] wangxj3 commented on issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
wangxj3 commented on issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525#issuecomment-844011242
>
>
> @wangxj3 Yeal, I also noticed this problem, this caused by each instance will listen the node path in zookeeper.
> ~One possible solution is to get the lock on zookeeper when insert the instance crash alert.~
> Can we think each instance should save at most one unsent instance alert in database?
---Can we think each instance should save at most one unsent instance alert in database?
Yes,this we will do something in alert .Do you have a dateil plan?
---And
The solution with lock on zookeeper can resove this issue.How ablut that,if we can get the time of server down ,insert this time to t_ds_alert ,this time is the PRIMARY KEY,so other server will fail to insert database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs closed issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
CalvinKirs closed issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #5525: server down will send repetitive message
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #5525:
URL: https://github.com/apache/dolphinscheduler/issues/5525#issuecomment-844403008
This is indeed a question that can be improved. Can anyone submit a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org