You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/10/14 04:45:01 UTC

[GitHub] [dolphinscheduler] yindarui123 opened a new pull request #6527: [Feature-#6422] task group queue function

yindarui123 opened a new pull request #6527:
URL: https://github.com/apache/dolphinscheduler/pull/6527


   # 1. the neccessity of task group queue
   Task group queue (TGQ) can achieve cross-project and cross-process concurrent control of tasks, reducing resource pressure on scheduling system or other big data cluster.
   TGQ also support priority-based control, which ensures that important tasks can be executed first. Users can also execute a task compulsively, ignoring the TGQ.
   # 2. the details about TGQ
   TGQ is essentially a flow limiter. By managing resources, the TGQ allows the tasks to obtain resources from the TGQ. In this way, the resources obtained by multiple tasks is limited and worker node's pressure is avoided. 
   The database optimistic lock is used to solve the thread safety problem in the distributed concurrent scenario. 
   Note that some tasks are not TGQ bound:
   1. The tasks that do not need to be performed by workers;
   2. The tasks which does not belongs to any TGQ; 
   3. The tasks is forcibly started by the user.
   ## 2.1 init a TGQ
   The user manually creates a TGQ. The size of the TGQ is specified by the user.
   ## 2.2 how does a TGQ works
   Each task configured with TGQ will apply for resource from the TGQ before being issued to the worker. If the TGQ has no available resources, the task will not be delivered to the worker, and the task wil wait for the resource release and ressend a request to TGQ.
   ## 2.3 recycle resources
   After receiving the response from the worker, TGQ will release the resources corresponding to the task.
   ## 2.4 fault tolerance
   In the distributed architecture, the fault tolerance mechanism is considerable. When the worker node is offline, the tasks with fault tolerance mechanism running on the worker node will be re-executed by the master. In order to prevent the same task from repeatedly applying for resources, when a task succeeds in applying for resources, it should check whether the task is already in the TGQ, and if so, it will resend the task to the worker. If not, allocate resources.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] dailidong commented on pull request #6527: [Feature-#6422] task group queue function

Posted by GitBox <gi...@apache.org>.
dailidong commented on pull request #6527:
URL: https://github.com/apache/dolphinscheduler/pull/6527#issuecomment-949402023


   thanks for your first contribution. good job 
   please only submit the related resources for this PR,  one PR just solves one issue  
   I think you will need my help, please contact me freely(mail: lidongdai@apache.org or wechat: taskflow), when added, please tell me your github id
   
   Welcome to join the DolphinScheduler Community!!!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng commented on pull request #6527: [Feature-#6422] Add Task group feature to limit the concurrency of taskInstance

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on pull request #6527:
URL: https://github.com/apache/dolphinscheduler/pull/6527#issuecomment-1004487080


   Hi @yindarui123 the TGQ feature had implemented, you can see #6722.  
   This PR will be closed,  and you can reopen it if need.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] CalvinKirs commented on pull request #6527: [Feature-#6422] task group queue function

Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #6527:
URL: https://github.com/apache/dolphinscheduler/pull/6527#issuecomment-947793766


   hi, please note that you have submitted a lot of irrelevant changes, you should only submit content that is relevant to you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng closed pull request #6527: [Feature-#6422] Add Task group feature to limit the concurrency of taskInstance

Posted by GitBox <gi...@apache.org>.
caishunfeng closed pull request #6527:
URL: https://github.com/apache/dolphinscheduler/pull/6527


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org