You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "ruanwenjun (via GitHub)" <gi...@apache.org> on 2024/04/19 15:10:46 UTC

[I] [DSIP-][Task] Deprecate DISPATCH status in task instance lifecycle [dolphinscheduler]

ruanwenjun opened a new issue, #15882:
URL: https://github.com/apache/dolphinscheduler/issues/15882

   ### Search before asking
   
   - [X] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.
   
   
   ### Motivation
   
   Once a TaskInstance is triggered, the status will be changed follow: submit -> dispatch -> running. 
   Submit means the task instance is initialized, dispatch means the task has been dispatch to worker, running means the task is running in worker. 
   
   <img width="1008" alt="image" src="https://github.com/apache/dolphinscheduler/assets/22415594/194ee04f-3307-4904-bd2c-ad86d08d69b8">
   
   
   The dispatch status is useful in the history design, since `dispatch` is a async operation, but in the latest implementation, the dispatch operation is a sync operation, once a task has dispatch to worker we can directly set the task instance status to running. This means we will also deprecate the waiting queue in worker, since the waiting queue doesn't help to manage the task, it just make the task lifecycle more difficult and make the resource management difficult, we hope the waiting task is all manage by master, once the task has been sent to worker, it will get thread to execute.
   
   
   ### Design Detail
   
   1. Remove the Dispatch status, once the task has been sent to worker then set task instance status to running.
   
   <img width="979" alt="image" src="https://github.com/apache/dolphinscheduler/assets/22415594/c870ab79-bad3-4278-88ad-f71e12eaf9cb">
   
   2. Deprecate the `TaskExecuteThreadsFullPolicy` in worker, the worker will reject the task once the worker thread pool is full.
   
   
   
   ### Compatibility, Deprecation, and Migration Plan
   
   This is compatibility, since the dispatch status will keep in code but only set it is depracated, will remove this after 3.3.0.
   
   ### Test Plan
   
   Tested by UT and E2E
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-29][Task] Deprecate DISPATCH status in task instance lifecycle [dolphinscheduler]

Posted by "LiJie20190102 (via GitHub)" <gi...@apache.org>.
LiJie20190102 commented on issue #15882:
URL: https://github.com/apache/dolphinscheduler/issues/15882#issuecomment-2840643495

   I am willing to do this !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-29][Task] Deprecate DISPATCH status in task instance lifecycle [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #15882:
URL: https://github.com/apache/dolphinscheduler/issues/15882#issuecomment-2840893222

   > I am willing to do this !
   
   Great, please provide your design detail, this issue will need to do many changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org