You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/09/29 09:41:22 UTC
[GitHub] [dolphinscheduler] yindarui123 opened a new pull request #6415: [Summer-2021] task group queue function 210290601
yindarui123 opened a new pull request #6415:
URL: https://github.com/apache/dolphinscheduler/pull/6415
# 1. the necessity of task group queue
Task group queue (TGQ) can achieve cross-project and cross-process concurrent control of tasks, reducing resource pressure on scheduling systems or other big data clusters.
TGQ also supports priority-based control, which ensures that important tasks can be executed first. Users can also execute a task compulsively, ignoring the TGQ.
# 2. the details about TGQ
TGQ is essentially a flow limiter. By managing resources, the TGQ allows the tasks to obtain resources from the TGQ. In this way, the resources obtained by multiple tasks is limited and the worker node's pressure is avoided.
The database optimistic lock is used to solve the thread-safety problem in the distributed concurrent scenario.
Note that some tasks are not TGQ bound:
- The tasks that do not need to be performed by workers;
- The tasks which do not belongs to any TGQ;
- The task is forcibly started by the user.
## 2.1 init a TGQ
The user manually creates a TGQ. The size of the TGQ is specified by the user.
## 2.2 how do a TGQ works
Each task configured with TGQ will apply for resources from the TGQ before being issued to the worker. If the TGQ has no available resources, the task will not be delivered to the worker, and the task will wait for the resource release and resend a request to TGQ.
## 2.3 recycle resources
After receiving the response from the worker, TGQ will release the resources corresponding to the task.
## 2.4 fault tolerance
In the distributed architecture, the fault tolerance mechanism is considerable. When the worker node is offline, the tasks with a fault tolerance mechanism running on the worker node will be re-executed by the master. In order to prevent the same task from repeatedly applying for resources, when a task succeeds in applying for resources, it should check whether the task is already in the TGQ, and if so, it will resend the task to the worker. If not, allocate resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs closed pull request #6415: [Summer-2021] task group queue function 210290601
Posted by GitBox <gi...@apache.org>.
CalvinKirs closed pull request #6415:
URL: https://github.com/apache/dolphinscheduler/pull/6415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on pull request #6415: [Summer-2021] task group queue function 210290601
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #6415:
URL: https://github.com/apache/dolphinscheduler/pull/6415#issuecomment-930063019
hi, thank you for your contribution. I noticed that you have modified a lot of files that you don’t want to close. I will close it. You can resubmit the PR after your modification is completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs closed pull request #6415: [Summer-2021] task group queue function 210290601
Posted by GitBox <gi...@apache.org>.
CalvinKirs closed pull request #6415:
URL: https://github.com/apache/dolphinscheduler/pull/6415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on pull request #6415: [Summer-2021] task group queue function 210290601
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #6415:
URL: https://github.com/apache/dolphinscheduler/pull/6415#issuecomment-930063019
hi, thank you for your contribution. I noticed that you have modified a lot of files that you don’t want to close. I will close it. You can resubmit the PR after your modification is completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org