You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/12 11:18:21 UTC
[GitHub] [flink] xintongsong opened a new pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
xintongsong opened a new pull request #13592:
URL: https://github.com/apache/flink/pull/13592
## What is the purpose of the change
This PR uses priority for mapping allocated containers to the requests, as a replacement for resource matching.
## Brief change log
- efabbbffff78d4c1600313ef784672aaa0068937: Introduce `TaskExecutorProcessSpecContainerResourcePriorityAdapter` for adapting between `TaskExecutorProcessSpec` and container `Resource` & `Priority`.
- 0217465ae215d990bcbc3a9cd2037e81d8b1067e: Switch from resource matching to priority matching.
## Verifying this change
- Add `TaskExecutorProcessSpecContainerResourcePriorityAdapterTest`
- Update `YarnResourceManagerDriverTest`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 69c41588b8273f1d95eece3643570adaf5288012 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594)
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r505186609
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -170,8 +159,12 @@ protected void initializeInternal() throws Exception {
resourceManagerClient.start();
final RegisterApplicationMasterResponse registerApplicationMasterResponse = registerApplicationMaster();
- getContainersFromPreviousAttempts(registerApplicationMasterResponse);
- updateMatchingStrategy(registerApplicationMasterResponse);
+ int maxPriority = getContainersFromPreviousAttempts(registerApplicationMasterResponse);
+ taskExecutorProcessSpecContainerResourcePriorityAdapter =
+ new TaskExecutorProcessSpecContainerResourcePriorityAdapter(
+ maxPriority + 1,
Review comment:
Should not be a problem. Yarn does not have constraints on the max priority value, as long as it does not grow lager than `Integer#MAX_VALUE`. In addition, we had this increasing priorities in Blink for years without any problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r505183845
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -228,21 +221,27 @@ public void deregisterApplication(ApplicationStatus finalStatus, @Nullable Strin
@Override
public CompletableFuture<YarnWorkerNode> requestResource(TaskExecutorProcessSpec taskExecutorProcessSpec) {
- final Optional<Resource> containerResourceOptional = getContainerResource(taskExecutorProcessSpec);
+ checkInitialized();
+
final CompletableFuture<YarnWorkerNode> requestResourceFuture = new CompletableFuture<>();
- if (containerResourceOptional.isPresent()) {
- resourceManagerClient.addContainerRequest(getContainerRequest(containerResourceOptional.get()));
+ final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.PriorityAndResource> priorityAndResourceOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getPriorityAndResource(taskExecutorProcessSpec);
+
+ if (!priorityAndResourceOpt.isPresent()) {
+ requestResourceFuture.completeExceptionally(
+ new ResourceManagerException(String.format("Could not compute the container Resource from the given TaskExecutorProcessSpec %s.", taskExecutorProcessSpec)));
Review comment:
There should be other logs from `taskExecutorProcessSpecContainerResourcePriorityAdapter` that gives the reasons why resource cannot be computed. For the moment, the only possible reason is that the requested resource is larger than the max container resource Yarn allows. See `TaskExecutorProcessSpecContainerResourcePriorityAdapter#tryAdaptResource` for the logs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0217465ae215d990bcbc3a9cd2037e81d8b1067e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437)
* 406dd5635a2995e909c417b8cbe9556f531494e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 406dd5635a2995e909c417b8cbe9556f531494e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 406dd5635a2995e909c417b8cbe9556f531494e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485)
* 69c41588b8273f1d95eece3643570adaf5288012 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong closed pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong closed pull request #13592:
URL: https://github.com/apache/flink/pull/13592
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 406dd5635a2995e909c417b8cbe9556f531494e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
}, {
"hash" : "755036cf9a0095e3d7b8e061ea2ffe4002b62ad3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7729",
"triggerID" : "755036cf9a0095e3d7b8e061ea2ffe4002b62ad3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686)
* 755036cf9a0095e3d7b8e061ea2ffe4002b62ad3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7729)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0217465ae215d990bcbc3a9cd2037e81d8b1067e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 69c41588b8273f1d95eece3643570adaf5288012 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0217465ae215d990bcbc3a9cd2037e81d8b1067e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437)
* 406dd5635a2995e909c417b8cbe9556f531494e0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-708277360
@KarmaGYZ Thanks for the review. Comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r505185009
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -260,36 +259,44 @@ public void releaseResource(YarnWorkerNode workerNode) {
// Internal
// ------------------------------------------------------------------------
- private void onContainersOfResourceAllocated(Resource resource, List<Container> containers) {
- final List<TaskExecutorProcessSpec> pendingTaskExecutorProcessSpecs =
- taskExecutorProcessSpecContainerResourceAdapter.getTaskExecutorProcessSpec(resource, matchingStrategy).stream()
- .flatMap(spec -> Collections.nCopies(getNumRequestedNotAllocatedWorkersFor(spec), spec).stream())
- .collect(Collectors.toList());
+ private void onContainersOfPriorityAllocated(Priority priority, List<Container> containers) {
+ final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.TaskExecutorProcessSpecAndResource> taskExecutorProcessSpecAndResourceOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getTaskExecutorProcessSpecAndResource(priority);
+
+ if (!taskExecutorProcessSpecAndResourceOpt.isPresent()) {
+ log.warn("Receive {} containers with unrecognized priority {}. This should not happen.",
+ containers.size(), priority.getPriority());
+ for (Container container : containers) {
+ returnExcessContainer(container);
+ }
+ return;
+ }
Review comment:
Actually I don't see any possible reasons that `taskExecutorProcessSpecAndResourceOpt` does not exist. This is just a safety net. Maybe we can just replace this with `checkState`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0217465ae215d990bcbc3a9cd2037e81d8b1067e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-708932219
@tillrohrmann Thanks for the review.
To answer your questions:
1. Yes, I think it is possible and makes sense. Please see my other reply for the details.
2. TBH, we do not have plan to use `allocationRequestId` at the moment. I think the priority based matching is an essential requirement for fine grained slot allocation, while the `allocationRequestId` base matching is an optimization for fine grained slot allocation with Hadoop 2.9+. Given the efforts and maintaining overhead needed for two sets of matching methods, I think we can worry about this optimization after the fine grained use cases coming true.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-709292366
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 406dd5635a2995e909c417b8cbe9556f531494e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485)
* 69c41588b8273f1d95eece3643570adaf5288012 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707617230
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0217465ae215d990bcbc3a9cd2037e81d8b1067e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659",
"triggerID" : "709292366",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686",
"triggerID" : "709678729",
"triggerType" : "MANUAL"
}, {
"hash" : "755036cf9a0095e3d7b8e061ea2ffe4002b62ad3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "755036cf9a0095e3d7b8e061ea2ffe4002b62ad3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7659) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7713) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7686)
* 755036cf9a0095e3d7b8e061ea2ffe4002b62ad3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r504365654
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/TaskExecutorProcessSpecContainerResourcePriorityAdapter.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn;
+
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Utility class for converting between Flink {@link TaskExecutorProcessSpec} and Yarn {@link Resource} and {@link Priority}.
+ */
+public class TaskExecutorProcessSpecContainerResourcePriorityAdapter {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TaskExecutorProcessSpecContainerResourcePriorityAdapter.class);
+
+ private final Map<TaskExecutorProcessSpec, Resource> taskExecutorProcessSpecToResource;
+ private final Map<TaskExecutorProcessSpec, Priority> taskExecutorProcessSpecToPriority;
Review comment:
We could merge these two maps. They should always have the same keyset by design.
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn;
+
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.junit.Test;
+
+import java.util.Collections;
+
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.not;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for {@link TaskExecutorProcessSpecContainerResourcePriorityAdapter}.
+ */
+public class TaskExecutorProcessSpecContainerResourcePriorityAdapterTest extends TestLogger {
+
+ private static final Resource MAX_CONTAINER_RESOURCE = Resource.newInstance(102400, 100);
Review comment:
I think we could add some tests for external resources. To be specific:
- Check could we construct `TaskExecutorProcessSpecContainerResourcePriorityAdapter` if the given external resource is not supported by the Yarn cluster.
- Under Hadoop 3.0+ or 2.10+, using `assumeTrue(HadoopUtils.isMinHadoopVersion(2, 10))`, set the external resource to `MAX_CONTAINER_RESOURCE` and add `testGetTaskExecutorProcessSpecWithExternalResource`.
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -260,36 +257,47 @@ public void releaseResource(YarnWorkerNode workerNode) {
// Internal
// ------------------------------------------------------------------------
- private void onContainersOfResourceAllocated(Resource resource, List<Container> containers) {
- final List<TaskExecutorProcessSpec> pendingTaskExecutorProcessSpecs =
- taskExecutorProcessSpecContainerResourceAdapter.getTaskExecutorProcessSpec(resource, matchingStrategy).stream()
- .flatMap(spec -> Collections.nCopies(getNumRequestedNotAllocatedWorkersFor(spec), spec).stream())
- .collect(Collectors.toList());
+ private void onContainersOfPriorityAllocated(Priority priority, List<Container> containers) {
+ final Optional<TaskExecutorProcessSpec> taskExecutorProcessSpecOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getTaskExecutorProcessSpec(priority);
+
+ if (!taskExecutorProcessSpecOpt.isPresent()) {
+ log.warn("Receive {} containers with unrecognized priority {}. This should not happen.",
+ containers.size(), priority.getPriority());
+ for (Container container : containers) {
+ returnExcessContainer(container);
+ }
+ return;
+ }
- int numPending = pendingTaskExecutorProcessSpecs.size();
- log.info("Received {} containers with resource {}, {} pending container requests.",
+ final TaskExecutorProcessSpec taskExecutorProcessSpec = taskExecutorProcessSpecOpt.get();
+
+ final Optional<Resource> resourceOpt = taskExecutorProcessSpecContainerResourcePriorityAdapter.getResource(taskExecutorProcessSpec);
+ Preconditions.checkState(resourceOpt.isPresent());
Review comment:
We could also make the value of `priorityToTaskExecutorProcessSpec` to a `Tuple2` of `TaskExecutorProcessSpec` and `Resource`.
##########
File path: flink-yarn/src/test/java/org/apache/flink/yarn/TaskExecutorProcessSpecContainerResourcePriorityAdapterTest.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.yarn;
+
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.hadoop.yarn.api.records.Priority;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.junit.Test;
+
+import java.util.Collections;
+
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.not;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for {@link TaskExecutorProcessSpecContainerResourcePriorityAdapter}.
+ */
+public class TaskExecutorProcessSpecContainerResourcePriorityAdapterTest extends TestLogger {
+
+ private static final Resource MAX_CONTAINER_RESOURCE = Resource.newInstance(102400, 100);
+
+ private static final TaskExecutorProcessSpec TASK_EXECUTOR_PROCESS_SPEC_1 = new TaskExecutorProcessSpec(
+ new CPUResource(1.0),
+ MemorySize.ofMebiBytes(100),
+ MemorySize.ofMebiBytes(100),
+ MemorySize.ofMebiBytes(100),
+ MemorySize.ofMebiBytes(100),
+ MemorySize.ofMebiBytes(100),
+ MemorySize.ofMebiBytes(100),
+ MemorySize.ofMebiBytes(100),
+ MemorySize.ofMebiBytes(100));
+
+ private static final TaskExecutorProcessSpec TASK_EXECUTOR_PROCESS_SPEC_2 = new TaskExecutorProcessSpec(
+ new CPUResource(2.0),
+ MemorySize.ofMebiBytes(200),
+ MemorySize.ofMebiBytes(200),
+ MemorySize.ofMebiBytes(200),
+ MemorySize.ofMebiBytes(200),
+ MemorySize.ofMebiBytes(200),
+ MemorySize.ofMebiBytes(200),
+ MemorySize.ofMebiBytes(200),
+ MemorySize.ofMebiBytes(200));
+
+ @Test
+ public void testGetResource() {
+ final TaskExecutorProcessSpecContainerResourcePriorityAdapter adapter = getAdapter();
+ final Resource resource = adapter.getResource(TASK_EXECUTOR_PROCESS_SPEC_1).get();
+ assertThat(resource.getMemory(), is(TASK_EXECUTOR_PROCESS_SPEC_1.getTotalProcessMemorySize().getMebiBytes()));
+ assertThat(resource.getVirtualCores(), is(TASK_EXECUTOR_PROCESS_SPEC_1.getCpuCores().getValue().intValue()));
+ }
+
+ @Test
+ public void testGetPriority() {
+ final TaskExecutorProcessSpecContainerResourcePriorityAdapter adapter = getAdapter();
+ final Priority priority1 = adapter.getPriority(TASK_EXECUTOR_PROCESS_SPEC_1).get();
+ final Priority priority2 = adapter.getPriority(TASK_EXECUTOR_PROCESS_SPEC_2).get();
+ final Priority priority3 = adapter.getPriority(TASK_EXECUTOR_PROCESS_SPEC_1).get();
+ assertThat(priority1, not(priority2));
+ assertThat(priority1, is(priority3));
+ }
+
+ @Test
+ public void testMaxContainerResource() {
+ final TaskExecutorProcessSpecContainerResourcePriorityAdapter adapter =
+ new TaskExecutorProcessSpecContainerResourcePriorityAdapter(
+ 1,
+ Resource.newInstance(100, 1),
+ Collections.emptyMap());
+ assertThat(adapter.getResource(TASK_EXECUTOR_PROCESS_SPEC_2).isPresent(), is(false));
+ assertThat(adapter.getPriority(TASK_EXECUTOR_PROCESS_SPEC_2).isPresent(), is(false));
Review comment:
It seems we could also use `getAdapter` here. We just need a `TASK_EXECUTOR_PROCESS_SPEC_3` with `CPUResource` 300.
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -349,32 +356,21 @@ private void startTaskExecutorInContainerAsync(
}, getMainThreadExecutor()));
}
- private Collection<AMRMClient.ContainerRequest> getPendingRequestsAndCheckConsistency(Resource resource, int expectedNum) {
- final Collection<Resource> equivalentResources = taskExecutorProcessSpecContainerResourceAdapter.getEquivalentContainerResource(resource, matchingStrategy);
- final List<? extends Collection<AMRMClient.ContainerRequest>> matchingRequests =
- equivalentResources.stream()
- .flatMap(equivalentResource -> resourceManagerClient.getMatchingRequests(
- RM_REQUEST_PRIORITY,
- ResourceRequest.ANY,
- equivalentResource).stream())
+ private Collection<AMRMClient.ContainerRequest> getPendingRequestsAndCheckConsistency(
+ Priority priority, Resource resource, int expectedNum) {
+ final List<AMRMClient.ContainerRequest> matchingRequests =
+ resourceManagerClient.getMatchingRequests(priority, ResourceRequest.ANY, resource)
+ .stream()
+ .flatMap(requests -> requests.stream())
Review comment:
```suggestion
.flatMap(Collection::stream)
```
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -260,36 +257,47 @@ public void releaseResource(YarnWorkerNode workerNode) {
// Internal
// ------------------------------------------------------------------------
- private void onContainersOfResourceAllocated(Resource resource, List<Container> containers) {
- final List<TaskExecutorProcessSpec> pendingTaskExecutorProcessSpecs =
- taskExecutorProcessSpecContainerResourceAdapter.getTaskExecutorProcessSpec(resource, matchingStrategy).stream()
- .flatMap(spec -> Collections.nCopies(getNumRequestedNotAllocatedWorkersFor(spec), spec).stream())
- .collect(Collectors.toList());
+ private void onContainersOfPriorityAllocated(Priority priority, List<Container> containers) {
+ final Optional<TaskExecutorProcessSpec> taskExecutorProcessSpecOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getTaskExecutorProcessSpec(priority);
+
+ if (!taskExecutorProcessSpecOpt.isPresent()) {
+ log.warn("Receive {} containers with unrecognized priority {}. This should not happen.",
+ containers.size(), priority.getPriority());
+ for (Container container : containers) {
+ returnExcessContainer(container);
+ }
+ return;
+ }
- int numPending = pendingTaskExecutorProcessSpecs.size();
- log.info("Received {} containers with resource {}, {} pending container requests.",
+ final TaskExecutorProcessSpec taskExecutorProcessSpec = taskExecutorProcessSpecOpt.get();
+
+ final Optional<Resource> resourceOpt = taskExecutorProcessSpecContainerResourcePriorityAdapter.getResource(taskExecutorProcessSpec);
+ Preconditions.checkState(resourceOpt.isPresent());
Review comment:
If the taskExecutorProcessSpecOpt is present, the resourceOpt should be present by design. I think we may have `taskExecutorProcessSpecContainerResourcePriorityAdapter#getTaskExecutorProcessSpecAndResource`, directly return an optional `Tuple2`. So, we do not need to call `taskExecutorProcessSpecContainerResourcePriorityAdapter.getResource(taskExecutorProcessSpec)` and check whether resourceOpt is present.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
}, {
"hash" : "69c41588b8273f1d95eece3643570adaf5288012",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594",
"triggerID" : "69c41588b8273f1d95eece3643570adaf5288012",
"triggerType" : "PUSH"
}, {
"hash" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 69c41588b8273f1d95eece3643570adaf5288012 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7594)
* 4dc5998fdfaf5c5c0e2b8fca0984d21ec508db7e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r505257420
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -228,21 +221,27 @@ public void deregisterApplication(ApplicationStatus finalStatus, @Nullable Strin
@Override
public CompletableFuture<YarnWorkerNode> requestResource(TaskExecutorProcessSpec taskExecutorProcessSpec) {
- final Optional<Resource> containerResourceOptional = getContainerResource(taskExecutorProcessSpec);
+ checkInitialized();
+
final CompletableFuture<YarnWorkerNode> requestResourceFuture = new CompletableFuture<>();
- if (containerResourceOptional.isPresent()) {
- resourceManagerClient.addContainerRequest(getContainerRequest(containerResourceOptional.get()));
+ final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.PriorityAndResource> priorityAndResourceOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getPriorityAndResource(taskExecutorProcessSpec);
+
+ if (!priorityAndResourceOpt.isPresent()) {
+ requestResourceFuture.completeExceptionally(
+ new ResourceManagerException(String.format("Could not compute the container Resource from the given TaskExecutorProcessSpec %s.", taskExecutorProcessSpec)));
Review comment:
True the logs exist. However, it might be not that easy to correlate these logging statements with the logging of the `ResourceManagerException` at a different place. If there is currently only one reason why the operation fails, maybe we can state it in the exception message as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r505196788
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -442,38 +444,25 @@ private RegisterApplicationMasterResponse registerApplicationMaster() throws Exc
return resourceManagerClient.registerApplicationMaster(rpcAddress, restPort, webInterfaceUrl);
}
- private void getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
+ private int getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
final List<Container> containersFromPreviousAttempts =
registerApplicationMasterResponseReflector.getContainersFromPreviousAttempts(registerApplicationMasterResponse);
final List<YarnWorkerNode> recoveredWorkers = new ArrayList<>();
log.info("Recovered {} containers from previous attempts ({}).", containersFromPreviousAttempts.size(), containersFromPreviousAttempts);
+ int maxPriority = 0;
for (Container container : containersFromPreviousAttempts) {
final YarnWorkerNode worker = new YarnWorkerNode(container, getContainerResourceId(container));
recoveredWorkers.add(worker);
+ maxPriority = Math.max(container.getPriority().getPriority(), maxPriority);
Review comment:
> I guess there is no easy way to calculate the TaskExecutorProcessSpec from a Container to restore the old priority to TaskExecutorProcessSpec mapping?
Unfortunately no.
> Would it be an option to say that the default TaskExecutorProcessSpec always has the priority 1? That way, the behaviour wrt priorities would not change for the current Flink version.
Good point. I think the easiest way is to always start the priority from `1` for a new attempt. Since there's only the default `TaskExecutorProcessSpec` for the current version, the priority should never increase within the attempt.
For each application, Yarn RM should only accept container requests from one AM at a time. Container requests from previous attempt will be discarded, only running containers can be recovered by the new AM. Therefore, there should be no problem the new attempt requesting containers with the same priorities the previous attempts have used.
I was trying to avoid using repeated priorities across attempts for better log readability. It's not absolutely necessary, because the container id already suggests which attempt the container is allocated. Since there's a good reason, I'm good with always start priority from 1 in a new attempt.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-709678729
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r505259078
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -442,38 +444,25 @@ private RegisterApplicationMasterResponse registerApplicationMaster() throws Exc
return resourceManagerClient.registerApplicationMaster(rpcAddress, restPort, webInterfaceUrl);
}
- private void getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
+ private int getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
final List<Container> containersFromPreviousAttempts =
registerApplicationMasterResponseReflector.getContainersFromPreviousAttempts(registerApplicationMasterResponse);
final List<YarnWorkerNode> recoveredWorkers = new ArrayList<>();
log.info("Recovered {} containers from previous attempts ({}).", containersFromPreviousAttempts.size(), containersFromPreviousAttempts);
+ int maxPriority = 0;
for (Container container : containersFromPreviousAttempts) {
final YarnWorkerNode worker = new YarnWorkerNode(container, getContainerResourceId(container));
recoveredWorkers.add(worker);
+ maxPriority = Math.max(container.getPriority().getPriority(), maxPriority);
Review comment:
Alright, this sounds good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r505258077
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -260,36 +259,44 @@ public void releaseResource(YarnWorkerNode workerNode) {
// Internal
// ------------------------------------------------------------------------
- private void onContainersOfResourceAllocated(Resource resource, List<Container> containers) {
- final List<TaskExecutorProcessSpec> pendingTaskExecutorProcessSpecs =
- taskExecutorProcessSpecContainerResourceAdapter.getTaskExecutorProcessSpec(resource, matchingStrategy).stream()
- .flatMap(spec -> Collections.nCopies(getNumRequestedNotAllocatedWorkersFor(spec), spec).stream())
- .collect(Collectors.toList());
+ private void onContainersOfPriorityAllocated(Priority priority, List<Container> containers) {
+ final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.TaskExecutorProcessSpecAndResource> taskExecutorProcessSpecAndResourceOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getTaskExecutorProcessSpecAndResource(priority);
+
+ if (!taskExecutorProcessSpecAndResourceOpt.isPresent()) {
+ log.warn("Receive {} containers with unrecognized priority {}. This should not happen.",
+ containers.size(), priority.getPriority());
+ for (Container container : containers) {
+ returnExcessContainer(container);
+ }
+ return;
+ }
Review comment:
Then I would vote for `checkState` to impose stricter conditions and to fail fast in case our assumptions are broken.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 406dd5635a2995e909c417b8cbe9556f531494e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
xintongsong commented on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707617522
cc @KarmaGYZ @tillrohrmann
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13592:
URL: https://github.com/apache/flink/pull/13592#issuecomment-707067008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7437",
"triggerID" : "0217465ae215d990bcbc3a9cd2037e81d8b1067e",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485",
"triggerID" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"triggerType" : "PUSH"
}, {
"hash" : "406dd5635a2995e909c417b8cbe9556f531494e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511",
"triggerID" : "707617230",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 406dd5635a2995e909c417b8cbe9556f531494e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7485) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7511)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13592: [FLINK-19324][yarn] Map requested and allocated containers with priority on YARN
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13592:
URL: https://github.com/apache/flink/pull/13592#discussion_r504657264
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -260,36 +259,44 @@ public void releaseResource(YarnWorkerNode workerNode) {
// Internal
// ------------------------------------------------------------------------
- private void onContainersOfResourceAllocated(Resource resource, List<Container> containers) {
- final List<TaskExecutorProcessSpec> pendingTaskExecutorProcessSpecs =
- taskExecutorProcessSpecContainerResourceAdapter.getTaskExecutorProcessSpec(resource, matchingStrategy).stream()
- .flatMap(spec -> Collections.nCopies(getNumRequestedNotAllocatedWorkersFor(spec), spec).stream())
- .collect(Collectors.toList());
+ private void onContainersOfPriorityAllocated(Priority priority, List<Container> containers) {
+ final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.TaskExecutorProcessSpecAndResource> taskExecutorProcessSpecAndResourceOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getTaskExecutorProcessSpecAndResource(priority);
+
+ if (!taskExecutorProcessSpecAndResourceOpt.isPresent()) {
+ log.warn("Receive {} containers with unrecognized priority {}. This should not happen.",
+ containers.size(), priority.getPriority());
+ for (Container container : containers) {
+ returnExcessContainer(container);
+ }
+ return;
+ }
Review comment:
Are we not failing hard here because this can happen in case of a JobManager failover?
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -442,38 +444,25 @@ private RegisterApplicationMasterResponse registerApplicationMaster() throws Exc
return resourceManagerClient.registerApplicationMaster(rpcAddress, restPort, webInterfaceUrl);
}
- private void getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
+ private int getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
final List<Container> containersFromPreviousAttempts =
registerApplicationMasterResponseReflector.getContainersFromPreviousAttempts(registerApplicationMasterResponse);
final List<YarnWorkerNode> recoveredWorkers = new ArrayList<>();
log.info("Recovered {} containers from previous attempts ({}).", containersFromPreviousAttempts.size(), containersFromPreviousAttempts);
+ int maxPriority = 0;
for (Container container : containersFromPreviousAttempts) {
final YarnWorkerNode worker = new YarnWorkerNode(container, getContainerResourceId(container));
recoveredWorkers.add(worker);
+ maxPriority = Math.max(container.getPriority().getPriority(), maxPriority);
Review comment:
I guess there is no easy way to calculate the `TaskExecutorProcessSpec` from a `Container` to restore the old priority to `TaskExecutorProcessSpec` mapping?
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -170,8 +159,12 @@ protected void initializeInternal() throws Exception {
resourceManagerClient.start();
final RegisterApplicationMasterResponse registerApplicationMasterResponse = registerApplicationMaster();
- getContainersFromPreviousAttempts(registerApplicationMasterResponse);
- updateMatchingStrategy(registerApplicationMasterResponse);
+ int maxPriority = getContainersFromPreviousAttempts(registerApplicationMasterResponse);
+ taskExecutorProcessSpecContainerResourcePriorityAdapter =
+ new TaskExecutorProcessSpecContainerResourcePriorityAdapter(
+ maxPriority + 1,
Review comment:
Will this be a problem that we continue increasing the priority value? Potentially, this can grow arbitrarily large.
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -442,38 +444,25 @@ private RegisterApplicationMasterResponse registerApplicationMaster() throws Exc
return resourceManagerClient.registerApplicationMaster(rpcAddress, restPort, webInterfaceUrl);
}
- private void getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
+ private int getContainersFromPreviousAttempts(final RegisterApplicationMasterResponse registerApplicationMasterResponse) {
final List<Container> containersFromPreviousAttempts =
registerApplicationMasterResponseReflector.getContainersFromPreviousAttempts(registerApplicationMasterResponse);
final List<YarnWorkerNode> recoveredWorkers = new ArrayList<>();
log.info("Recovered {} containers from previous attempts ({}).", containersFromPreviousAttempts.size(), containersFromPreviousAttempts);
+ int maxPriority = 0;
for (Container container : containersFromPreviousAttempts) {
final YarnWorkerNode worker = new YarnWorkerNode(container, getContainerResourceId(container));
recoveredWorkers.add(worker);
+ maxPriority = Math.max(container.getPriority().getPriority(), maxPriority);
Review comment:
Would it be an option to say that the default `TaskExecutorProcessSpec` always has the priority `1`? That way, the behaviour wrt priorities would not change for the current Flink version.
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManagerDriver.java
##########
@@ -228,21 +221,27 @@ public void deregisterApplication(ApplicationStatus finalStatus, @Nullable Strin
@Override
public CompletableFuture<YarnWorkerNode> requestResource(TaskExecutorProcessSpec taskExecutorProcessSpec) {
- final Optional<Resource> containerResourceOptional = getContainerResource(taskExecutorProcessSpec);
+ checkInitialized();
+
final CompletableFuture<YarnWorkerNode> requestResourceFuture = new CompletableFuture<>();
- if (containerResourceOptional.isPresent()) {
- resourceManagerClient.addContainerRequest(getContainerRequest(containerResourceOptional.get()));
+ final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.PriorityAndResource> priorityAndResourceOpt =
+ taskExecutorProcessSpecContainerResourcePriorityAdapter.getPriorityAndResource(taskExecutorProcessSpec);
+
+ if (!priorityAndResourceOpt.isPresent()) {
+ requestResourceFuture.completeExceptionally(
+ new ResourceManagerException(String.format("Could not compute the container Resource from the given TaskExecutorProcessSpec %s.", taskExecutorProcessSpec)));
Review comment:
Would a bit more details be helpful here? Maybe why we could not compute the container resource?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org