You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/04/08 06:25:01 UTC
[GitHub] [dolphinscheduler] JinyLeeChina opened a new issue, #9401: [Bug] [Master] Master crazy print log
JinyLeeChina opened a new issue, #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
**1. No worker:**
[INFO] 2022-04-08 06:14:58.688 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread:[309] - process event: State Event :key: null type: TASK_STATE_CHANGE executeStatus: NEED_FAULT_TOLERANCE task instance id: 8 process instance id: 7 context: null
**2. Worker high load**
fail to execute : ... due to no suitable worker, current task needs worker group ... to execute
**3. The running task was deleted**
[INFO] 2022-04-06 11:13:46.749 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread:[301] - process event: State Event :key: null type: TASK_TIMEOUT executeStatus: null task instance id: 10277 process instance id: 6 context: null
[ERROR] 2022-04-06 11:13:46.749 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread:[264] - state handle error:
java.lang.NullPointerException: null
### What you expected to happen
Master should not print logs frequently.
### How to reproduce
Please take a look
### Anything else
_No response_
### Version
2.0.5
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] czeming commented on issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
czeming commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1098292630
Through communication, I learned the current situation:
The task itself must be executed successfully, so it needs to be retried continuously. If there is a problem when pushing it to `worker`, the log must also be output.
But I think outputting logs actually means some kind of feedback or recording problems. I speculate that adding status to `task` itself may solve the problem.
For example, `waiting` and `pending`, `waiting` means that the `task` instance is newly created, and `pending` means that the pre execution logic is satisfied and attempts to push it to `worker`.
I suggest updating from `waiting` to `pending` and pushing the task to `worker` when preparing to push the task, so that users can realize that the task has met the execution conditions.
If the `task` in `waiting` status fails to be pushed to `worker` for the first time, the error log can be printed and updated to `pending` status. After that, try again and again. If the `task` in the `pending` state fails to push, the log will not be output. When it succeeds, switch to the current execution state and output the corresponding success log.
Of course, how the underlying method of this logic call controls the log needs to be discussed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] JinyLeeChina closed issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
JinyLeeChina closed issue #9401: [Bug] [Master] Master frequent print logs
URL: https://github.com/apache/dolphinscheduler/issues/9401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1127097590
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1099794743
> > Through communication, I learned the current situation: The task itself must be executed successfully, so it needs to be retried continuously. If there is a problem when pushing it to `worker`, the log must also be output.
> > But I think outputting logs actually means some kind of feedback or recording problems. I speculate that adding status to `task` itself may solve the problem. For example, `waiting` and `pending`, `waiting` means that the `task` instance is newly created, and `pending` means that the pre execution logic is satisfied and attempts to push it to `worker`.
> > I suggest updating from `waiting` to `pending` and pushing the task to `worker` when preparing to push the task, so that users can realize that the task has met the execution conditions. If the `task` in `waiting` status fails to be pushed to `worker` for the first time, the error log can be printed and updated to `pending` status. After that, try again and again. If the `task` in the `pending` state fails to push, the log will not be output. When it succeeds, switch to the current execution state and output the corresponding success log.
> > Of course, how the underlying method of this logic call controls the log needs to be discussed.
>
> I think i can make this simple to understand. The solution is to add the status of task instances to replace the log output.
>
> 1. Add `waiting suitable worker` state to represent no suitable worker log output.
> 2. Add `waiing for the worker's load to decrease` state to represent worker in high load.
> 3. Add `stop due to the running task was deleted` state to represent the running task was deleted.
>
> After adding these status to the task instance. We may delete the irrelevant log to prevent frequent log printing. So that the user can understand task instance state instead of locate the problem by querying the logs. And this solution is also applicable to other places where logs are frequently printed.
>
> WDYT? @caishunfeng @lenboo @JinyLeeChina
>
> Related issue: #7516
Good idea, but I think just add `waitting suitable worker` state for dispatch is OK, because `fail to execute : ... due to no suitable worker, current task needs worker group ... to execute` print the most.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1098639210
> Through communication, I learned the current situation: The task itself must be executed successfully, so it needs to be retried continuously. If there is a problem when pushing it to `worker`, the log must also be output.
>
> But I think outputting logs actually means some kind of feedback or recording problems. I speculate that adding status to `task` itself may solve the problem. For example, `waiting` and `pending`, `waiting` means that the `task` instance is newly created, and `pending` means that the pre execution logic is satisfied and attempts to push it to `worker`.
>
> I suggest updating from `waiting` to `pending` and pushing the task to `worker` when preparing to push the task, so that users can realize that the task has met the execution conditions. If the `task` in `waiting` status fails to be pushed to `worker` for the first time, the error log can be printed and updated to `pending` status. After that, try again and again. If the `task` in the `pending` state fails to push, the log will not be output. When it succeeds, switch to the current execution state and output the corresponding success log.
>
> Of course, how the underlying method of this logic call controls the log needs to be discussed.
I think i can make this simple to understand. The solution is to add the status of task instances to replace the log output.
1. Add ```waiting suitable worker``` state to represent no suitable worker log output.
2. Add ```waiing for the worker's load to decrease``` state to represent worker in high load.
3. Add ```stop due to the running task was deleted``` state to represent the running task was deleted.
After adding these status to the task instance. We may delete the irrelevant log to prevent frequent log printing. So that the user can understand task instance state instead of locate the problem by querying the logs. And this solution is also applicable to other places where logs are frequently printed.
WDYT? @caishunfeng @lenboo @JinyLeeChina
Related issue: #7516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1159316410
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] czeming commented on issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
czeming commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1097810831
I want to try to solve this problem.
At present, after the execution of an unknown exception fails, it will actually be caught and will not execute `remove` and keep trying again. This should be the reason for the frequent output of the same log (multiple executions of the same data still fail).
It is suggested to use `poll` to ensure the deletion of data, but I am not sure whether there is any consideration of retry here.
There are two options:
1. Directly use `poll`. If it fails, execute the failure processing
2. Use `poll` and try again for a certain number of times, such as `while`. If it fails, execute the failure processing.
Failure handling can also be divided into two types:
1. Output log directly
2. Establish a failure queue globally, print the log and add the failure data to the failure queue.
If failure queue is adopted, the function of failure queue needs to be discussed.
======
I found a problem with the existing code:
`org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread#handleEvents` called `remove`
`org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread#stateEventHandler` called `remove`
In fact, when result = true, the call will be repeated and `remove` will traverse the queue once.
Using peek to read and execute time-consuming tasks may cause concurrent misreading.
For the above reasons, `poll` is recommended.
@JinyLeeChina For the specific problems you encounter, you need more detailed stack information to troubleshoot. I've seen the logic of execution. Except for `process_state_change`, there are few abnormal logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #9401: [Bug] [Master] Master crazy print log
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1092484294
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #9401: [Bug] [Master] Master frequent print logs
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #9401:
URL: https://github.com/apache/dolphinscheduler/issues/9401#issuecomment-1099806074
> > > Through communication, I learned the current situation: The task itself must be executed successfully, so it needs to be retried continuously. If there is a problem when pushing it to `worker`, the log must also be output.
> > > But I think outputting logs actually means some kind of feedback or recording problems. I speculate that adding status to `task` itself may solve the problem. For example, `waiting` and `pending`, `waiting` means that the `task` instance is newly created, and `pending` means that the pre execution logic is satisfied and attempts to push it to `worker`.
> > > I suggest updating from `waiting` to `pending` and pushing the task to `worker` when preparing to push the task, so that users can realize that the task has met the execution conditions. If the `task` in `waiting` status fails to be pushed to `worker` for the first time, the error log can be printed and updated to `pending` status. After that, try again and again. If the `task` in the `pending` state fails to push, the log will not be output. When it succeeds, switch to the current execution state and output the corresponding success log.
> > > Of course, how the underlying method of this logic call controls the log needs to be discussed.
> >
> >
> > I think i can make this simple to understand. The solution is to add the status of task instances to replace the log output.
> >
> > 1. Add `waiting suitable worker` state to represent no suitable worker log output.
> > 2. Add `waiing for the worker's load to decrease` state to represent worker in high load.
> > 3. Add `stop due to the running task was deleted` state to represent the running task was deleted.
> >
> > After adding these status to the task instance. We may delete the irrelevant log to prevent frequent log printing. So that the user can understand task instance state instead of locate the problem by querying the logs. And this solution is also applicable to other places where logs are frequently printed.
> > WDYT? @caishunfeng @lenboo @JinyLeeChina
> > Related issue: #7516
>
> Good idea, but I think just add `waitting suitable worker` state for dispatch is OK, because `fail to execute : ... due to no suitable worker, current task needs worker group ... to execute` print the most.
Worker high load will also be output frequently in the production environment with more workflows, such as my production environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org