You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "inventertom (via GitHub)" <gi...@apache.org> on 2023/08/07 07:37:49 UTC

[GitHub] [dolphinscheduler] inventertom opened a new issue, #14710: [Bug] [Module Name] The master log is growing too fast

inventertom opened a new issue, #14710:
URL: https://github.com/apache/dolphinscheduler/issues/14710

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What happened
   
   background:2 masters, 4 workers, high availability of master settings
   A few days ago, a master down machine, after the reboot the master log grew rapidly, and the disk quickly filled up
   log :
   [ERROR] 2023-08-07 13:19:48.670 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[278] - [WorkflowInstance-4053240][TaskInstance-9596384] - State event handle error, will retry this event: StateEvent(key=4053244-0-4053240-9596384, type=TASK_STATE_CHANGE, executionStatus=RUNNING_EXECUTION, taskInstanceId=9596384, taskCode=0, processInstanceId=4053240, context=null, channel=null)
   org.apache.dolphinscheduler.server.master.event.StateEventHandleException: Task state event handle error, due to the task is not in activeTaskProcessorMaps
           at org.apache.dolphinscheduler.server.master.event.TaskStateEventHandler.handleStateEvent(TaskStateEventHandler.java:84)
           at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable.handleEvents(WorkflowExecuteRunnable.java:270)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   [ERROR] 2023-08-07 13:19:48.670 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[278] - [WorkflowInstance-4053601][TaskInstance-9600083] - State event handle error, will retry this event: StateEvent(key=4053603-0-4053601-9600083, type=TASK_STATE_CHANGE, executionStatus=RUNNING_EXECUTION, taskInstanceId=9600083, taskCode=0, processInstanceId=4053601, context=null, channel=null)
   org.apache.dolphinscheduler.server.master.event.StateEventHandleException: Task state event handle error, due to the task is not in activeTaskProcessorMaps
           at org.apache.dolphinscheduler.server.master.event.TaskStateEventHandler.handleStateEvent(TaskStateEventHandler.java:84)
           at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable.handleEvents(WorkflowExecuteRunnable.java:270)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   
   
   We tried the method
   1. Modify the ID status of the task instance to Failed,Restart the master of the node, and the log is still full quickly
   2. reference  #11918  ,  Delete the task instance that corresponds to the id of the task instance  ,  Restart the master of the node, and the log is still full quickly
   
   
   
   
   
   ### What you expected to happen
   
   The log is printed normally, do not grow too fast, causing the disk to fill up
   
   ### How to reproduce
   
   Set the highly available master, pre-task workflow, and then let the master down machine resume
   
   ### Anything else
   
   _No response_
   
   ### Version
   
   3.1.x
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] inventertom closed issue #14710: [Bug] [Module Name] The master log is growing too fast

Posted by "inventertom (via GitHub)" <gi...@apache.org>.
inventertom closed issue #14710: [Bug] [Module Name] The master log is growing too fast
URL: https://github.com/apache/dolphinscheduler/issues/14710


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org