You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "ruanwenjun (via GitHub)" <gi...@apache.org> on 2025/11/20 01:56:19 UTC

[I] [DSIP-94][Worker] Supports setting separate maximum concurrency limits for different task types on Workers. [dolphinscheduler]

ruanwenjun opened a new issue, #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707

   ### Search before asking
   
   - [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP.
   
   
   ### Motivation
   
   _No response_
   
   ### Design Detail
   
   _No response_
   
   ### Compatibility, Deprecation, and Migration Plan
   
   _No response_
   
   ### Test Plan
   
   _No response_
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-94][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3575808202

   > TaskGroup can also be used to control the concurrency of task. This new feature may have some conflict with the existing TaskGroup
   
   TaskGroup feels rather peculiar to use. In most cases, we use worker groups to partition resources and manage concurrency. It is difficult to enforce users to set `TaskGroups` for tasks, but tasks must utilize worker groups and workers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-94][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3574280965

   TaskGroup can also be used to control the concurrent number of task instances. This new feature may have some conflict with the existing TaskGroup


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-94][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3570954450

   > Can you confirm if my understanding is correct? For example, suppose we have worker1 and worker2, and worker1 has more resources (such as CPU and memory) compared to worker2.
   > 
   > In such an unbalanced cluster environment, I can set different concurrency limits for worker1 and worker2. For instance, on worker1, the concurrency for shell tasks can be set to 10, and for SQL tasks, it can be set to 5. Meanwhile, since worker2 has fewer resources, the concurrency for shell tasks on worker2 can be set to 5, and for SQL tasks, it can be set to 2.
   
   Yes, this is a use case. However, our worker node configurations are typically identical. We aim to support varying levels of concurrency for different tasks—for instance, SQL/HTTP tasks can handle higher concurrency, while shell tasks support lower concurrency.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-98][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "det101 (via GitHub)" <gi...@apache.org>.
det101 commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3669952440

   > For example, like this?
   > 
   > <img alt="Image" width="1906" height="647" src="https://private-user-images.githubusercontent.com/15990723/518553605-e5c4aeb0-fdfb-4509-92a5-cd37ed3ea4a3.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjYwNTk1MTgsIm5iZiI6MTc2NjA1OTIxOCwicGF0aCI6Ii8xNTk5MDcyMy81MTg1NTM2MDUtZTVjNGFlYjAtZmRmYi00NTA5LTkyYTUtY2QzN2VkM2VhNGEzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTEyMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMjE4VDEyMDAxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFiMzA0YzI4MTc1MmI1MDEwNjc4ZTgxYzk1OTAxOTNmYmY2MDJiZjlhYWQ5MDkzODJmMDA1YjBlNWM5MzY4MjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Qdj5MQdWlntch5IqhwXb9mHG2XNWsuDTQ_ryHp8hZhg"> <img alt="Image" width="847" height="731" src="https://private-user-images.githubusercontent.com/15990723/518553606-0875e5ad-954a-4a0e-aa09-5ff4691d91f3.png?jwt=eyJ
 0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjYwNTk1MTgsIm5iZiI6MTc2NjA1OTIxOCwicGF0aCI6Ii8xNTk5MDcyMy81MTg1NTM2MDYtMDg3NWU1YWQtOTU0YS00YTBlLWFhMDktNWZmNDY5MWQ5MWYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTEyMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMjE4VDEyMDAxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTczMTE5OGNkNjMxYTRkNmNjNGNkODQ1ZjMzY2JjYTMxNzNhMGZhYmNjMmE0MmM2MmUwZTBkMTQxYzc2OThhODMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.1c17rxxhVIoYIMoARXNvS4CV074lnku4zQcwNg4fflM">
   
   How to assign default values


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-94][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3574591426

   For example, like this?
   
   <img width="1906" height="647" alt="Image" src="https://github.com/user-attachments/assets/e5c4aeb0-fdfb-4509-92a5-cd37ed3ea4a3" />
   <img width="847" height="731" alt="Image" src="https://github.com/user-attachments/assets/0875e5ad-954a-4a0e-aa09-5ff4691d91f3" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-98][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3673383043

   > > For example, like this?
   > > <img alt="Image" width="1906" height="647" src="https://private-user-images.githubusercontent.com/15990723/518553605-e5c4aeb0-fdfb-4509-92a5-cd37ed3ea4a3.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjYwNTk1MTgsIm5iZiI6MTc2NjA1OTIxOCwicGF0aCI6Ii8xNTk5MDcyMy81MTg1NTM2MDUtZTVjNGFlYjAtZmRmYi00NTA5LTkyYTUtY2QzN2VkM2VhNGEzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTEyMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMjE4VDEyMDAxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFiMzA0YzI4MTc1MmI1MDEwNjc4ZTgxYzk1OTAxOTNmYmY2MDJiZjlhYWQ5MDkzODJmMDA1YjBlNWM5MzY4MjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.Qdj5MQdWlntch5IqhwXb9mHG2XNWsuDTQ_ryHp8hZhg"> <img alt="Image" width="847" height="731" src="https://private-user-images.githubusercontent.com/15990723/518553606-0875e5ad-954a-4a0e-aa09-5ff4691d91f3.png?jwt=e
 yJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjYwNTk1MTgsIm5iZiI6MTc2NjA1OTIxOCwicGF0aCI6Ii8xNTk5MDcyMy81MTg1NTM2MDYtMDg3NWU1YWQtOTU0YS00YTBlLWFhMDktNWZmNDY5MWQ5MWYzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTEyMTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMjE4VDEyMDAxOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTczMTE5OGNkNjMxYTRkNmNjNGNkODQ1ZjMzY2JjYTMxNzNhMGZhYmNjMmE0MmM2MmUwZTBkMTQxYzc2OThhODMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.1c17rxxhVIoYIMoARXNvS4CV074lnku4zQcwNg4fflM">
   > 
   > How to assign default values
   
   I don't have a clear idea now. Maybe it can be placed in application.yaml just like the configuration of load balancing. 
   For example:
   ```
   worker:
     # worker host weight to dispatch tasks, default value 100
     host-weight: 100
     shell-task-max-limit: 4
     sql-task-max-limit: 100
      ...
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-98][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3677856104

   @Mrhs121 
   The config may look like below
   ```yaml
   worker:
       server-load-protection:
           task-load-protection:
             - type: SHELL
               max-running-task-instances: 20
            - type: SQL
               - datasource-type: MYSQL
                  max-running-task-instances: 100
               - datasource-type: ORACLE
                  max-running-task-instances: 100
   ```
   Each task plugin can have its own protection logic.
   
   Dynamic modification of configuration parameters is not required at this stage and will be addressed in a unified manner in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-94][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "Zzih (via GitHub)" <gi...@apache.org>.
Zzih commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3574221970

   I think this feature is great. Our tasks are mostly Seatunnel + SQL, and we need to control Seatunnel concurrency without limiting SQL concurrency.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-94][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "Zzih (via GitHub)" <gi...@apache.org>.
Zzih commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3574309942

   > TaskGroup can also be used to control the concurrency of task. This new feature may have some conflict with the existing TaskGroup
   
   It can be extended based on the existing task group functionality.The existing task group functionality is tied to the project, and I don't understand why it's designed this way. I think it's a bad design.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [DSIP-94][Worker] Add per-task-type maximum concurrency limits on Workers [dolphinscheduler]

Posted by "Mrhs121 (via GitHub)" <gi...@apache.org>.
Mrhs121 commented on issue #17707:
URL: https://github.com/apache/dolphinscheduler/issues/17707#issuecomment-3569378888

   Can you confirm if my understanding is correct? For example, suppose we have worker1 and worker2, and worker1 has more resources (such as CPU and memory) compared to worker2.
   
   In such an unbalanced cluster environment, I can set different concurrency limits for worker1 and worker2. For instance, on worker1, the concurrency for shell tasks can be set to 10, and for SQL tasks, it can be set to 5. Meanwhile, since worker2 has fewer resources, the concurrency for shell tasks on worker2 can be set to 5, and for SQL tasks, it can be set to 2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org