You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "RocMarshal (via GitHub)" <gi...@apache.org> on 2023/02/11 14:16:45 UTC
[GitHub] [incubator-streampark] RocMarshal opened a new issue, #2324: [Feature] Introduce yarn queue setting to manage the queue resouce
RocMarshal opened a new issue, #2324:
URL: https://github.com/apache/incubator-streampark/issues/2324
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
# Intention
If this is meaningful for improving usability in `yarn-application` & `yarn-session` execution mode,
IMO, Maybe we could introduce the queue-resource-manage feature in the long term. After that, There will be some advantage:
- Keeping the available yarn-queues in `streampark` is helpful to keep the boundary between `Streampark` and `yarn-cluster`.
- Prevent queue abuse.
- Prevent failed submission caused by non-existent queue.
- Make it easier for users to perceive and to user the available yarn queues.
- Support select queue when users using the `yarn-application` & `yarn-session` mode to deploy the jobs.
# What
- The expected outline of the yarn queue management
<img width="1788" alt="image" src="https://user-images.githubusercontent.com/64569824/218262226-ef26a472-533d-4d35-a7d4-3a788000397f.png">
<img width="1787" alt="image" src="https://user-images.githubusercontent.com/64569824/218262245-446ce66e-255c-48f3-bcd8-c592cdcc6d5e.png">
- Support select queue when deploying on yarn.
deploy yarn session
<img width="1774" alt="image" src="https://user-images.githubusercontent.com/64569824/218262661-81fa1e19-f24a-41da-a34f-49bb751ebd92.png">
deploy yarn-application jobs
<img width="1738" alt="image" src="https://user-images.githubusercontent.com/64569824/218262745-da2f19d8-5a67-427e-a0d2-f1bcde6813b1.png">
# How
- Introduce the yarn queue table to store the queue info
fields:
- id
- team_id
- queue_label
- description
- create_time
- modify_time
- Need to note: Check the queue references( by yarn-session clusters & yarn-application jobs) when delete yarn queue.
### Usage Scenario
_No response_
### Related issues
#2299
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on issue #2324: [Feature] Introduce yarn queue setting to manage the queue resouce
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on issue #2324:
URL: https://github.com/apache/incubator-streampark/issues/2324#issuecomment-1475291146
https://github.com/apache/incubator-streampark/pull/2467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on issue #2324: [Feature] Introduce yarn queue setting to manage the queue resouce
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on issue #2324:
URL: https://github.com/apache/incubator-streampark/issues/2324#issuecomment-1455096475
For compatibility issues, we only verify the queue label permission of the hostParams attribute, and ignore the queue label parameter verification at the dynamic parameter level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on issue #2324: [Feature] Introduce yarn queue setting to manage the queue resouce
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on issue #2324:
URL: https://github.com/apache/incubator-streampark/issues/2324#issuecomment-1441701348
The interface preview for FE
```
@Slf4j
@Validated
@RestController
@RequestMapping("flink/yarnQueue")
public class YarnQueueController {
@Autowired private YarnQueueService yarnQueueService;
/** Pageful request the queues in the teamId */
@PostMapping("list")
public RestResponse list(RestRequest request, Long teamId) {
return RestResponse.success(IPage<YarnQueue>);
}
/**
* Check for the yarn queue if exists in the team or the queueLabel format is invalid.
* status msg
* 0 Success
* 1 The queue already existed in the team
* 2 The queueLabel format in invalid.
*/
@PostMapping("check")
public RestResponse check(Long teamId, YarnQueue yarnQueue) {
ResponseResult checkResult;
return RestResponse.success(checkResult);
}
@PostMapping("create")
@RequiresPermissions("yarnQueue:create")
public RestResponse create(YarnQueue yarnQueue) {
return RestResponse.success(true/false);
}
@PostMapping("update")
@RequiresPermissions("yarnQueue:update")
public RestResponse update(YarnQueue yarnQueue) {
return RestResponse.success();
}
@PostMapping("get")
public RestResponse get(Long queueId) throws InternalException {
YarnQueue yarnQueue;
return RestResponse.success(yarnQueue);
}
@PostMapping("delete")
@RequiresPermissions("yarnQueue:delete")
public RestResponse delete(YarnQueue yarnQueue) {
return RestResponse.success();
}
}
}
```
Hi, @wangsizhu0504 I make the draft of interfaces you would use. Please take a look if you have free time. Thank you~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal closed issue #2324: [Feature] Introduce yarn queue setting to manage the queue resouce
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal closed issue #2324: [Feature] Introduce yarn queue setting to manage the queue resouce
URL: https://github.com/apache/incubator-streampark/issues/2324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org