You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/08/16 06:30:45 UTC
[GitHub] [incubator-seatunnel] Hisoka-X opened a new issue, #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Hisoka-X opened a new issue, #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
### Purpose Of Design
1. It can provide the resource address required for the execute of the Job
2. Ability to dynamically scale nodes in third-party resource management services
3. Each Worker can support running multiple tasks of different Jobs to make full use of resources
4. Worker can recover in time if abnormality occurs
### Resource Definition
#### Unit resources
````java
public class Worker {
private Slot[] slots;
}
````
````java
public class Slot {
private Resource resource;
}
````
#### Unit resource definition
The number of resources each unit slot should have
````java
public class Resource {
private Memory memorySize;
// only work on kubernetes
private CPU cpu;
}
````
SeaTunnel supports normal worker memory allocation, and if running on Kubernetes, it can also support allocation for the CPU running the worker.
### Implementation process
#### Standalone
1. The Master and Worker are started, the Worker registers the information to the ResourceManager, and the status is monitored by regular heartbeat.
2. JobMaster requests resources from ResourceManager
3. If the resources are sufficient, the corresponding Worker information will be provided to the JobMaster. If not enough, the JobMaster will not have enough resources.
4. After the JobMaster receives the feedback, it determines whether to start scheduling the task or throw an exception
#### Third Party Resource Manager
1. The Master and Worker are started, the Worker registers the information to the ResourceManager, and the status is monitored by regular heartbeat.
2. JobMaster requests resources from ResourceManager
3. If the resources are sufficient, provide the corresponding Worker information to the JobMaster, if not, apply for resources from the third-party resource management and start the Worker.
4. Worker registers with ResourceManager
5. After the JobMaster receives the feedback, it determines whether to start scheduling the task or throw an exception
6. Return resources to ResourceManager after JobMaster execution ends
7. According to the policy, the ResourceManager determines whether the worker needs to be released and then returned to the third-party resource management system or reserved for the next task.
#### The specific process is as follows:
### Worker's slot allocation
Each Worker needs to allocate its resources to different Slots for the Job to run and use. How to split the Worker into different Slots?
There are two allocation strategies:
1. When the Worker starts, the average distribution is completed according to the configuration information
2. Dynamic allocation. When a task applies for resources, the corresponding slot is dynamically created. The size of the slot is determined according to the number of resources required by the task. This can maximize the use of resources and improve the success rate of scheduling large-resource tasks.
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1227210304
> public class Resource {
> private Memory memorySize;
> // only work on kubernetes
> private CPU cpu;
>}
Does standalone use this `Resource` too? If I want run 500 task in a node(this is possible because of shared threads), How do we define the number of slots supported by a node？And, how many `CPU` and `memorySize` a slot use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1216398468
How do we achieve memory isolation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1227254665
> > public class Resource {
> > private Memory memorySize;
> > // only work on kubernetes
> > private CPU cpu;
> > }
>
> Does standalone use this `Resource` too? If I want run 500 task in a node(this is possible because of shared threads), How do we define the number of slots supported by a node？And, how many `CPU` and `memorySize` a slot use?
First, standalone yes. The number of our slots is configurable, such as 1024M of memory, 1024 slots can be declared (if you want to). It means that you can run 1024 tasks at the same time, and each slot has 1M memory resources (declarative, not occupied. Later, consider implementing occupied resource application in off-heap memory). By the way, we can share one slot with multi-task in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
hailin0 commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1217629341
How do we achieve requested resource are locked, or release after a task running crash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X closed issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
Hisoka-X closed issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
URL: https://github.com/apache/incubator-seatunnel/issues/2426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1217744218
> How do we achieve requested resource are locked, or release after a task running crash
Locked by Resource Manager, Release by JobMaster and tell Resource Manager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1216206632
In my view, I perfer call `Slot` as `Grid`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1216370016
> Dynamic allocation. When a task applies for resources, the corresponding slot is dynamically created. The size of the slot is determined according to the number of resources required by the task. This can maximize the use of resources and improve the success rate of scheduling large-resource tasks.
Yes, I think `Dynamic allocation` is better. Our `TaskExecutionService` can run task in shared thread, So I think the `TaskExecutionService` can run more tasks if every task only have a small dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1216405604
> How do we achieve memory isolation?
There is no real isolation, only logical occupation. In fact, other tasks on the same worker can exceed his memory limit. This is for the fact that we only use heap memory at present. If we use out-of-heap memory in the future, we can complete isolation in the true sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on issue #2426: [ST-Engine][Design] Resource Manager design from seatunnel engine
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on issue #2426:
URL: https://github.com/apache/incubator-seatunnel/issues/2426#issuecomment-1216203003
If you have better designs or suggestions, please leave a message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org