You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/03/08 08:54:57 UTC

[GitHub] [flink] KarmaGYZ opened a new pull request #15112: [FLINK-21480][coordination] Respect external resources from resource requirements

KarmaGYZ opened a new pull request #15112:
URL: https://github.com/apache/flink/pull/15112


   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   Respect external resources from resource requirements
   
   ## Brief change log
   
   *(for example:)*
     - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact*
     - *Deployments RPC transmits only the blob storage reference*
     - *TaskManagers retrieve the TaskInfo from the blob cache*
   
   
   ## Verifying this change
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (100MB)*
     - *Extended integration test for recovery after master (JobManager) failure*
     - *Added test that validates that TaskInfo is transferred only once across recoveries*
     - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
     - The serializers: (yes / no / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / no / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
     - The S3 file system connector: (yes / no / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] KarmaGYZ commented on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-800102934


   Thanks for the review @xintongsong . I've verified it in a Kubernetes cluster.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dba07460fc5b5c5fe89c357b2eeb776ab60d58bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396) 
   * 0d505494b6fc2f31dcad47706fa20714812f1bc6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dba07460fc5b5c5fe89c357b2eeb776ab60d58bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] KarmaGYZ commented on a change in pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on a change in pull request #15112:
URL: https://github.com/apache/flink/pull/15112#discussion_r591120661



##########
File path: flink-core/src/main/java/org/apache/flink/api/common/operators/ResourceSpec.java
##########
@@ -369,14 +373,21 @@ public Builder setManagedMemoryMB(int managedMemoryMB) {
             return this;
         }
 
-        public Builder setGPUResource(double gpus) {
-            this.gpuResource = new GPUResource(gpus);
+        public Builder addExtendedResource(String name, Resource extendedResource) {

Review comment:
       The External Reource Framework allows user to set any `resourceName` for GPUs. A `GPUResource` would break that assumption. However, I think that shortcut would enhance usability. We can do it in the future.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d505494b6fc2f31dcad47706fa20714812f1bc6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] xintongsong closed pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
xintongsong closed pull request #15112:
URL: https://github.com/apache/flink/pull/15112


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ea2c71fb676eaab22727a7ba440bda308215cf9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ea2c71fb676eaab22727a7ba440bda308215cf9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] xintongsong commented on a change in pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #15112:
URL: https://github.com/apache/flink/pull/15112#discussion_r590131518



##########
File path: flink-core/src/main/java/org/apache/flink/api/common/operators/ResourceSpec.java
##########
@@ -93,7 +92,7 @@ private ResourceSpec(
             final MemorySize taskHeapMemory,
             final MemorySize taskOffHeapMemory,
             final MemorySize managedMemory,
-            final Resource... extendedResources) {
+            final Map<String, Resource> extendedResources) {

Review comment:
       It would be nice to enforce `ExternalResource` rather than `Resource` here.
   
   IIUC, what prevent us from doing so is that, return types of `Resource#merge/subtract/multiply/divide/create` are hard coded to `Resource`. We might consider to solve this with generics. That should also help replacing `Resource` with `CPUResource` wherever needed.
   
   ```
   abstract class Resource<T extends Resource<T>> {
       T merge(T other);
   }
   
   class ExternalResource extends Resource<ExternalResource> {}
   ```

##########
File path: flink-core/src/main/java/org/apache/flink/api/common/operators/ResourceSpec.java
##########
@@ -369,14 +373,21 @@ public Builder setManagedMemoryMB(int managedMemoryMB) {
             return this;
         }
 
-        public Builder setGPUResource(double gpus) {
-            this.gpuResource = new GPUResource(gpus);
+        public Builder addExtendedResource(String name, Resource extendedResource) {

Review comment:
       I would be fine with providing a shortcut setter for a commonly used external resource like GPU. That would also save us from touching all the test cases.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
##########
@@ -118,13 +141,22 @@ public boolean equals(Object obj) {
                     && Objects.equals(this.taskHeapSize, that.taskHeapSize)
                     && Objects.equals(this.taskOffHeapSize, that.taskOffHeapSize)
                     && Objects.equals(this.networkMemSize, that.networkMemSize)
-                    && Objects.equals(this.managedMemSize, that.managedMemSize);
+                    && Objects.equals(this.managedMemSize, that.managedMemSize)
+                    && Objects.equals(this.extendedResources, that.extendedResources);
         }
         return false;
     }
 
     @Override
     public String toString() {
+        final StringBuilder extendedResourceStr = new StringBuilder(extendedResources.size() * 10);
+        for (Map.Entry<String, Resource> resource : extendedResources.entrySet()) {
+            extendedResourceStr
+                    .append(", ")
+                    .append(resource.getKey())
+                    .append('=')
+                    .append(resource.getValue().getValue());
+        }

Review comment:
       Consider deduplicating this string generation with a util method. We can put it in `ExternalResourceUtils`.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerUtils.java
##########
@@ -29,14 +29,24 @@
      */
     public static ResourceProfile generateDefaultSlotResourceProfile(
             WorkerResourceSpec workerResourceSpec, int numSlotsPerWorker) {
-        return ResourceProfile.newBuilder()
-                .setCpuCores(workerResourceSpec.getCpuCores().divide(numSlotsPerWorker))
-                .setTaskHeapMemory(workerResourceSpec.getTaskHeapSize().divide(numSlotsPerWorker))
-                .setTaskOffHeapMemory(
-                        workerResourceSpec.getTaskOffHeapSize().divide(numSlotsPerWorker))
-                .setManagedMemory(workerResourceSpec.getManagedMemSize().divide(numSlotsPerWorker))
-                .setNetworkMemory(workerResourceSpec.getNetworkMemSize().divide(numSlotsPerWorker))
-                .build();
+        final ResourceProfile.Builder resourceProfileBuilder =

Review comment:
       I would suggest to add external resources to `WorkerResourceSpec` and `TaskExecutorResourceSpec` in one commit, so that this util is always consistent with `TaskExecutorResourceUtils` in the git history.

##########
File path: flink-core/src/main/java/org/apache/flink/api/common/operators/ResourceSpec.java
##########
@@ -369,14 +373,21 @@ public Builder setManagedMemoryMB(int managedMemoryMB) {
             return this;
         }
 
-        public Builder setGPUResource(double gpus) {
-            this.gpuResource = new GPUResource(gpus);
+        public Builder addExtendedResource(String name, Resource extendedResource) {
+            this.extendedResources.put(name, extendedResource);
+            return this;
+        }
+
+        public Builder addExtendedResources(Map<String, Resource> extendedResources) {

Review comment:
       We should not need the map keys for the argument. Same for `ResourceProfile.Builder`.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
##########
@@ -174,9 +208,26 @@ public Builder setManagedMemoryMB(int managedMemoryMB) {
             return this;
         }
 
+        public Builder addExtendedResource(String name, Resource extendedResource) {
+            this.extendedResources.put(name, extendedResource);
+            return this;
+        }
+
+        public Builder addExtendedResources(Map<String, Resource> extendedResources) {
+            if (extendedResources != null) {
+                this.extendedResources.putAll(extendedResources);
+            }
+            return this;
+        }

Review comment:
       Same here:
   * `Resource` -> `ExternalResource`
   * Should not need the names.

##########
File path: flink-core/src/main/java/org/apache/flink/api/common/operators/ResourceSpec.java
##########
@@ -369,14 +373,21 @@ public Builder setManagedMemoryMB(int managedMemoryMB) {
             return this;
         }
 
-        public Builder setGPUResource(double gpus) {
-            this.gpuResource = new GPUResource(gpus);
+        public Builder addExtendedResource(String name, Resource extendedResource) {

Review comment:
       We should not need the argument `name`. Same for `ResourceProfile.Builder`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] KarmaGYZ commented on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-796473222


   @xintongsong Thanks for the review! PR updated.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 928bf6ee7c0b59a07dcb9f6125b9e45b66df1338 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666) 
   * 53d315fa94babbbbe29b5f7e8b8d5a61d496371e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 928bf6ee7c0b59a07dcb9f6125b9e45b66df1338 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 928bf6ee7c0b59a07dcb9f6125b9e45b66df1338 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666) 
   * 53d315fa94babbbbe29b5f7e8b8d5a61d496371e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769) 
   * 601d6cca0f61145903eecb7432d8b45f35e621c6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795",
       "triggerID" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "39dac6ca02d80777ad67f2ed1b337a596192f633",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "39dac6ca02d80777ad67f2ed1b337a596192f633",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e302f61027624a13d1209a49fa8afbc28774649",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14869",
       "triggerID" : "5e302f61027624a13d1209a49fa8afbc28774649",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 601d6cca0f61145903eecb7432d8b45f35e621c6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795) 
   * 39dac6ca02d80777ad67f2ed1b337a596192f633 UNKNOWN
   * 5e302f61027624a13d1209a49fa8afbc28774649 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14869) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dba07460fc5b5c5fe89c357b2eeb776ab60d58bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396) 
   * 0d505494b6fc2f31dcad47706fa20714812f1bc6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795",
       "triggerID" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 53d315fa94babbbbe29b5f7e8b8d5a61d496371e Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769) 
   * 601d6cca0f61145903eecb7432d8b45f35e621c6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d505494b6fc2f31dcad47706fa20714812f1bc6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429) 
   * 928bf6ee7c0b59a07dcb9f6125b9e45b66df1338 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792607838


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit 0ea2c71fb676eaab22727a7ba440bda308215cf9 (Mon Mar 08 09:16:57 UTC 2021)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795",
       "triggerID" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "39dac6ca02d80777ad67f2ed1b337a596192f633",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "39dac6ca02d80777ad67f2ed1b337a596192f633",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e302f61027624a13d1209a49fa8afbc28774649",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5e302f61027624a13d1209a49fa8afbc28774649",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 601d6cca0f61145903eecb7432d8b45f35e621c6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795) 
   * 39dac6ca02d80777ad67f2ed1b337a596192f633 UNKNOWN
   * 5e302f61027624a13d1209a49fa8afbc28774649 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795",
       "triggerID" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "39dac6ca02d80777ad67f2ed1b337a596192f633",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "39dac6ca02d80777ad67f2ed1b337a596192f633",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 601d6cca0f61145903eecb7432d8b45f35e621c6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795) 
   * 39dac6ca02d80777ad67f2ed1b337a596192f633 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] xintongsong commented on a change in pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #15112:
URL: https://github.com/apache/flink/pull/15112#discussion_r594812450



##########
File path: flink-core/src/main/java/org/apache/flink/api/common/resources/Resource.java
##########
@@ -30,7 +30,8 @@
 
 /** Base class for resources one can specify. */
 @Internal
-public abstract class Resource implements Serializable, Comparable<Resource> {
+public abstract class Resource<T extends Resource<T>>

Review comment:
       nit: Convert `other` to `T` rather than `Resource` in `equals()`

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/TaskExecutorProcessSpec.java
##########
@@ -97,7 +105,8 @@ public TaskExecutorProcessSpec(
             MemorySize networkMemSize,
             MemorySize managedMemorySize,
             MemorySize jvmMetaspaceSize,
-            MemorySize jvmOverheadSize) {
+            MemorySize jvmOverheadSize,
+            Map<String, ExternalResource> extendedResources) {

Review comment:
       I'd suggest replace `Map<String, ExternalResource>` with `Collection<ExternalResource>`.
   The problem of the current implementation is that, we rely on the caller to guarantee the key of the entry equals the name of the value external resource.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
##########
@@ -197,14 +226,30 @@ public Builder setNumSlots(int numSlots) {
             return this;
         }
 
+        public Builder addExtendedResource(ExternalResource extendedResource) {
+            this.extendedResources.put(extendedResource.getName(), extendedResource);
+            return this;
+        }
+
+        public Builder addExtendedResources(Collection<ExternalResource> extendedResources) {
+            if (extendedResources != null) {
+                extendedResources.forEach(
+                        extendedResource ->
+                                this.extendedResources.put(
+                                        extendedResource.getName(), extendedResource));
+            }
+            return this;
+        }

Review comment:
       Same here for existing external resource names.

##########
File path: flink-core/src/main/java/org/apache/flink/api/common/operators/ResourceSpec.java
##########
@@ -368,14 +373,24 @@ public Builder setManagedMemoryMB(int managedMemoryMB) {
             return this;
         }
 
-        public Builder setGPUResource(double gpus) {
-            this.gpuResource = new GPUResource(gpus);
+        public Builder addExtendedResource(ExternalResource extendedResource) {
+            this.extendedResources.put(extendedResource.getName(), extendedResource);
+            return this;
+        }
+
+        public Builder addExtendedResources(Collection<ExternalResource> extendedResources) {
+            if (extendedResources != null) {
+                extendedResources.forEach(
+                        extendedResource ->
+                                this.extendedResources.put(
+                                        extendedResource.getName(), extendedResource));
+            }

Review comment:
       Same for the `ResourceProfile.Builder`.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/externalresource/ExternalResourceUtils.java
##########
@@ -157,6 +159,29 @@ private ExternalResourceUtils() {
         return externalResourceAmountMap;
     }
 
+    /** Get the map of resource name and Resources of all of enabled external resources. */
+    public static Map<String, ExternalResource> getExternalResourcesMap(Configuration config) {
+        return getExternalResourceAmountMap(config).entrySet().stream()
+                .collect(
+                        Collectors.toMap(
+                                Map.Entry::getKey,
+                                entry -> new ExternalResource(entry.getKey(), entry.getValue())));
+    }
+
+    /** Generate the string expression of the given external resources. */
+    public static String generateExternalResourcesString(
+            Map<String, ExternalResource> extendedResources) {
+        final StringBuilder extendedResourceStr = new StringBuilder(extendedResources.size() * 10);
+        for (Map.Entry<String, ExternalResource> resource : extendedResources.entrySet()) {
+            extendedResourceStr
+                    .append(", ")
+                    .append(resource.getKey())
+                    .append('=')
+                    .append(resource.getValue().getValue());
+        }
+        return extendedResourceStr.toString();
+    }

Review comment:
       It's against intuition that the external resources string starts with a `, `.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/TaskExecutorProcessSpec.java
##########
@@ -97,7 +105,8 @@ public TaskExecutorProcessSpec(
             MemorySize networkMemSize,
             MemorySize managedMemorySize,
             MemorySize jvmMetaspaceSize,
-            MemorySize jvmOverheadSize) {
+            MemorySize jvmOverheadSize,
+            Map<String, ExternalResource> extendedResources) {

Review comment:
       Same for `WorkerResourceSpec` and `TaskExecutorResourceSpec`.

##########
File path: flink-core/src/main/java/org/apache/flink/api/common/operators/ResourceSpec.java
##########
@@ -368,14 +373,24 @@ public Builder setManagedMemoryMB(int managedMemoryMB) {
             return this;
         }
 
-        public Builder setGPUResource(double gpus) {
-            this.gpuResource = new GPUResource(gpus);
+        public Builder addExtendedResource(ExternalResource extendedResource) {
+            this.extendedResources.put(extendedResource.getName(), extendedResource);
+            return this;
+        }
+
+        public Builder addExtendedResources(Collection<ExternalResource> extendedResources) {
+            if (extendedResources != null) {
+                extendedResources.forEach(
+                        extendedResource ->
+                                this.extendedResources.put(
+                                        extendedResource.getName(), extendedResource));
+            }

Review comment:
       It's unclear to me what is the expected behavior by contract if an external resource already exist with the same name as the to be added resource.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 928bf6ee7c0b59a07dcb9f6125b9e45b66df1338 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666) 
   * 53d315fa94babbbbe29b5f7e8b8d5a61d496371e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ea2c71fb676eaab22727a7ba440bda308215cf9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293) 
   * dba07460fc5b5c5fe89c357b2eeb776ab60d58bb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ea2c71fb676eaab22727a7ba440bda308215cf9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0d505494b6fc2f31dcad47706fa20714812f1bc6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429) 
   * 928bf6ee7c0b59a07dcb9f6125b9e45b66df1338 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14396",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14429",
       "triggerID" : "0d505494b6fc2f31dcad47706fa20714812f1bc6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14666",
       "triggerID" : "928bf6ee7c0b59a07dcb9f6125b9e45b66df1338",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14769",
       "triggerID" : "53d315fa94babbbbe29b5f7e8b8d5a61d496371e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795",
       "triggerID" : "601d6cca0f61145903eecb7432d8b45f35e621c6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 601d6cca0f61145903eecb7432d8b45f35e621c6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14795) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] xintongsong commented on a change in pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #15112:
URL: https://github.com/apache/flink/pull/15112#discussion_r595681050



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceProfile.java
##########
@@ -640,15 +642,18 @@ public Builder setNetworkMemoryMB(int networkMemoryMB) {
             return this;
         }
 
-        public Builder addExtendedResource(String name, Resource extendedResource) {
-            this.extendedResources.put(name, extendedResource);
+        // Add the given extended resource, the old value with the same resource name will be
+        // override if present.
+        public Builder setExtendedResource(Resource extendedResource) {
+            this.extendedResources.put(extendedResource.getName(), extendedResource);
             return this;
         }
 
-        public Builder addExtendedResources(Map<String, Resource> extendedResources) {
-            if (extendedResources != null) {
-                this.extendedResources.putAll(extendedResources);
-            }
+        // Add the given extended resources, this will override all the previous records.

Review comment:
       ```
   Add the given extended resources. This will discard all the previous added extended resources.
   ```

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/TaskExecutorProcessSpec.java
##########
@@ -109,18 +120,27 @@ public TaskExecutorProcessSpec(
                         networkMemSize,
                         managedMemorySize),
                 new JvmMetaspaceAndOverhead(jvmMetaspaceSize, jvmOverheadSize),
-                1);
+                1,
+                extendedResources);
     }
 
     protected TaskExecutorProcessSpec(
             CPUResource cpuCores,
             TaskExecutorFlinkMemory flinkMemory,
             JvmMetaspaceAndOverhead jvmMetaspaceAndOverhead,
-            int numSlots) {
+            int numSlots,
+            Collection<ExternalResource> extendedResources) {
 
         super(flinkMemory, jvmMetaspaceAndOverhead);
         this.cpuCores = cpuCores;
         this.numSlots = numSlots;
+        this.extendedResources =
+                Preconditions.checkNotNull(extendedResources).stream()
+                        .filter(resource -> !resource.isZero())
+                        .collect(Collectors.toMap(ExternalResource::getName, Function.identity()));
+        Preconditions.checkState(

Review comment:
       Same for `WorkerResourceSpec` and `TaskExecutorResourceSpec`.

##########
File path: flink-core/src/main/java/org/apache/flink/api/common/resources/Resource.java
##########
@@ -51,50 +52,51 @@ protected Resource(String name, BigDecimal value) {
         this.value = value;
     }
 
-    public Resource merge(Resource other) {
+    public T merge(T other) {
         checkNotNull(other, "Cannot merge with null resources");
         checkArgument(getClass() == other.getClass(), "Merge with different resource type");
-        checkArgument(name.equals(other.name), "Merge with different resource name");
+        checkArgument(name.equals(other.getName()), "Merge with different resource name");
 
-        return create(value.add(other.value));
+        return create(value.add(other.getValue()));
     }
 
-    public Resource subtract(Resource other) {
+    public T subtract(T other) {
         checkNotNull(other, "Cannot subtract null resources");
         checkArgument(getClass() == other.getClass(), "Minus with different resource type");
-        checkArgument(name.equals(other.name), "Minus with different resource name");
+        checkArgument(name.equals(other.getName()), "Minus with different resource name");
         checkArgument(
-                value.compareTo(other.value) >= 0,
+                value.compareTo(other.getValue()) >= 0,
                 "Try to subtract a larger resource from this one.");
 
-        return create(value.subtract(other.value));
+        return create(value.subtract(other.getValue()));
     }
 
-    public Resource multiply(BigDecimal multiplier) {
+    public T multiply(BigDecimal multiplier) {
         return create(value.multiply(multiplier));
     }
 
-    public Resource multiply(int multiplier) {
+    public T multiply(int multiplier) {
         return multiply(BigDecimal.valueOf(multiplier));
     }
 
-    public Resource divide(BigDecimal by) {
+    public T divide(BigDecimal by) {
         return create(value.divide(by, 16, RoundingMode.DOWN));
     }
 
-    public Resource divide(int by) {
+    public T divide(int by) {
         return divide(BigDecimal.valueOf(by));
     }
 
     @Override
+    @SuppressWarnings("unchecked")

Review comment:
       It would be nice to keep the scope of `@SuppressWarnings` as small as possible. In this case, the single statement `T other = (T) o;` should be enough.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/TaskExecutorProcessSpec.java
##########
@@ -109,18 +120,27 @@ public TaskExecutorProcessSpec(
                         networkMemSize,
                         managedMemorySize),
                 new JvmMetaspaceAndOverhead(jvmMetaspaceSize, jvmOverheadSize),
-                1);
+                1,
+                extendedResources);
     }
 
     protected TaskExecutorProcessSpec(
             CPUResource cpuCores,
             TaskExecutorFlinkMemory flinkMemory,
             JvmMetaspaceAndOverhead jvmMetaspaceAndOverhead,
-            int numSlots) {
+            int numSlots,
+            Collection<ExternalResource> extendedResources) {
 
         super(flinkMemory, jvmMetaspaceAndOverhead);
         this.cpuCores = cpuCores;
         this.numSlots = numSlots;
+        this.extendedResources =
+                Preconditions.checkNotNull(extendedResources).stream()
+                        .filter(resource -> !resource.isZero())
+                        .collect(Collectors.toMap(ExternalResource::getName, Function.identity()));
+        Preconditions.checkState(

Review comment:
       `checkArgument` is preferred.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceProfile.java
##########
@@ -640,15 +642,18 @@ public Builder setNetworkMemoryMB(int networkMemoryMB) {
             return this;
         }
 
-        public Builder addExtendedResource(String name, Resource extendedResource) {
-            this.extendedResources.put(name, extendedResource);
+        // Add the given extended resource, the old value with the same resource name will be
+        // override if present.
+        public Builder setExtendedResource(Resource extendedResource) {
+            this.extendedResources.put(extendedResource.getName(), extendedResource);
             return this;
         }
 
-        public Builder addExtendedResources(Map<String, Resource> extendedResources) {
-            if (extendedResources != null) {
-                this.extendedResources.putAll(extendedResources);
-            }
+        // Add the given extended resources, this will override all the previous records.
+        public Builder setExtendedResources(Collection<Resource> extendedResources) {
+            this.extendedResources =
+                    extendedResources.stream()
+                            .collect(Collectors.toMap(Resource::getName, Function.identity()));

Review comment:
       JavaDoc style is preferred. It hints the compiler/IDE that there's a binding between the docs and the following class/field/method.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceProfile.java
##########
@@ -485,15 +486,11 @@ public String toString() {
             return "ResourceProfile{ANY}";
         }
 
-        final StringBuilder extendedResourceStr = new StringBuilder(extendedResources.size() * 10);
-        for (Map.Entry<String, ExternalResource> resource : extendedResources.entrySet()) {
-            extendedResourceStr
-                    .append(", ")
-                    .append(resource.getKey())
-                    .append('=')
-                    .append(resource.getValue().getValue());
-        }
-        return "ResourceProfile{" + getResourceString() + extendedResourceStr + '}';
+        return "ResourceProfile{"
+                + getResourceString()
+                + ", "
+                + ExternalResourceUtils.generateExternalResourcesString(extendedResources)

Review comment:
       What if there's no external resource? We will have something like `ResourceProfile{a=b, c=d, }`.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceProfile.java
##########
@@ -485,15 +486,11 @@ public String toString() {
             return "ResourceProfile{ANY}";
         }
 
-        final StringBuilder extendedResourceStr = new StringBuilder(extendedResources.size() * 10);
-        for (Map.Entry<String, ExternalResource> resource : extendedResources.entrySet()) {
-            extendedResourceStr
-                    .append(", ")
-                    .append(resource.getKey())
-                    .append('=')
-                    .append(resource.getValue().getValue());
-        }
-        return "ResourceProfile{" + getResourceString() + extendedResourceStr + '}';
+        return "ResourceProfile{"
+                + getResourceString()
+                + ", "
+                + ExternalResourceUtils.generateExternalResourcesString(extendedResources)

Review comment:
       Same for `TaskExecutorProcessSpec` and `WorkerResourceSpec`.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/externalresource/ExternalResourceUtils.java
##########
@@ -46,12 +49,19 @@
 
     private static final Logger LOG = LoggerFactory.getLogger(ExternalResourceUtils.class);
 
+    // Indicate an empty external resources list
+    public static final String EMPTY = "empty-list";

Review comment:
       I'd suggest to:
   - Use the phrase `none`
   - Add the constant in `ExternalResourceOptions`
   - Mention in the description of `EXTERNAL_RESOURCE_LIST` that the keyword is preserved

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/externalresource/ExternalResourceUtils.java
##########
@@ -157,6 +167,30 @@ private ExternalResourceUtils() {
         return externalResourceAmountMap;
     }
 
+    /** Get the collection of all enabled external resources. */
+    public static Collection<ExternalResource> getExternalResourcesCollection(
+            Configuration config) {
+        return getExternalResourceAmountMap(config).entrySet().stream()
+                .map(entry -> new ExternalResource(entry.getKey(), entry.getValue()))
+                .collect(Collectors.toList());
+    }
+
+    /** Generate the string expression of the given external resources. */
+    public static String generateExternalResourcesString(
+            Map<String, ExternalResource> extendedResources) {
+        final StringBuilder extendedResourceStr = new StringBuilder(extendedResources.size() * 10);
+        for (Map.Entry<String, ExternalResource> resource : extendedResources.entrySet()) {
+            if (extendedResourceStr.length() > 0) {
+                extendedResourceStr.append(", ");
+            }
+            extendedResourceStr
+                    .append(resource.getKey())
+                    .append('=')
+                    .append(resource.getValue().getValue());
+        }

Review comment:
       This can be simplified as follows:
   ```
   return extendedResources.entrySet().stream()
           .map(e -> e.getKey() + "=" + e.getValue().getValue())
           .collect(Collectors.joining(", "));
   ```

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceProfile.java
##########
@@ -640,15 +642,18 @@ public Builder setNetworkMemoryMB(int networkMemoryMB) {
             return this;
         }
 
-        public Builder addExtendedResource(String name, Resource extendedResource) {
-            this.extendedResources.put(name, extendedResource);
+        // Add the given extended resource, the old value with the same resource name will be
+        // override if present.

Review comment:
       ```
   Add the given extended resource. The old value with the same resource name will be replaced if present.
   ```

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/externalresource/ExternalResourceUtils.java
##########
@@ -157,6 +167,30 @@ private ExternalResourceUtils() {
         return externalResourceAmountMap;
     }
 
+    /** Get the collection of all enabled external resources. */
+    public static Collection<ExternalResource> getExternalResourcesCollection(
+            Configuration config) {
+        return getExternalResourceAmountMap(config).entrySet().stream()
+                .map(entry -> new ExternalResource(entry.getKey(), entry.getValue()))
+                .collect(Collectors.toList());
+    }
+
+    /** Generate the string expression of the given external resources. */
+    public static String generateExternalResourcesString(
+            Map<String, ExternalResource> extendedResources) {

Review comment:
       `Map<String, ExternalResource>` -> `Collection<ExternalResource>`

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/types/ResourceProfile.java
##########
@@ -640,15 +642,18 @@ public Builder setNetworkMemoryMB(int networkMemoryMB) {
             return this;
         }
 
-        public Builder addExtendedResource(String name, Resource extendedResource) {
-            this.extendedResources.put(name, extendedResource);
+        // Add the given extended resource, the old value with the same resource name will be
+        // override if present.
+        public Builder setExtendedResource(Resource extendedResource) {
+            this.extendedResources.put(extendedResource.getName(), extendedResource);
             return this;
         }
 
-        public Builder addExtendedResources(Map<String, Resource> extendedResources) {
-            if (extendedResources != null) {
-                this.extendedResources.putAll(extendedResources);
-            }
+        // Add the given extended resources, this will override all the previous records.
+        public Builder setExtendedResources(Collection<Resource> extendedResources) {
+            this.extendedResources =
+                    extendedResources.stream()
+                            .collect(Collectors.toMap(Resource::getName, Function.identity()));

Review comment:
       Same for `ResourceSpec.Builder` and `WorkerResourceSpec.Builder`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #15112: [FLINK-21480][runtime] Respect external resources from resource requirements

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15112:
URL: https://github.com/apache/flink/pull/15112#issuecomment-792620617


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293",
       "triggerID" : "0ea2c71fb676eaab22727a7ba440bda308215cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dba07460fc5b5c5fe89c357b2eeb776ab60d58bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0ea2c71fb676eaab22727a7ba440bda308215cf9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14293) 
   * dba07460fc5b5c5fe89c357b2eeb776ab60d58bb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org