You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/03/05 11:02:38 UTC

[GitHub] [flink] xintongsong opened a new pull request #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.

xintongsong opened a new pull request #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320

## What is the purpose of the change

This is the first step of FLINK-14106, including all the major changes inside SlotManager and changes to the RM/SM interfaces, except changes for metrics and status.

At the end of this step, SlotManager should allocate resource from ResourceManager with a WorkerResourceSpec, instead of slot ResourceProfile. At this step, the WorkerResourceSpec will not be used, and the active RMs will always use `ActiveResourceManager#taskExecutorProcessSpec` for requesting TMs. We will change that in subsequent steps.

## Brief change log

- e42d87754e15c3baed9258f29d91ff9719283759..82f9a631caadd9033f30cbc9229f1e6e0facfd72: Minor code clean-ups.
- 14881da7d062e986e20ac548249a42457dd4f7f0: Introduce `WorkerResourceSpec`.
- c3778a58f6d9440e3df540a19e69019734f38d9e: Create `SlotManagerImpl` with default `WorkerResourceSpec` in active resource manager setups.
- 66f4d52b8d767d053e72cbf25da51d5831e87b78: Create SlotManagerImpl with default numSlotsPerWorker.
- cd2b13e45b4cd84f7688a20b26142a382f56423a: Compute pending slot profiles inside SlotManager when allocating resource.
- This means also check unfulfillable slot profiles inside SlotManager.
- 484543c14aa9ce5a0d510e131ff6d354f4eacf25: ResourceManager retrieve a collection of pending workers from SlotManager, instead of number of pending slots.
- 76829127134c8f10fdf7a87097638915e2859b2e: Remove numSlotsPerTaskManager from ActiveResourceManager and ContaineredTaskManagerParameters.
- Since now we compute pending slot profiles inside SlotManager, ResourceManagers no longer need to be aware of number of slots per worker.
- cb785eb68e49927f24ada564772be2aa8e2a9cfd: SlotManager allocate resource from ResourceManager with WorkerRequest instead of ResourceProfile.
- This marks ResourceManager is no longer aware of slot ResourceProfiles.

## Verifying this change

This is a refactoring work. Most of the behaviors are already covered by exist test cases. Only added a few new test cases for the added data structures and logics.
- Add `WorkerResourceSpecTest`.
- Add test case in `TaskExecutorProcessUtilsTest`.
- Add test case in `SlotManagerImplTest`.

## Does this pull request potentially affect one of the following parts:

- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes)
- The S3 file system connector: (no)

## Documentation

- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-608491962
 
 
   I think the last part of my comment was answered by #11353.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388697084
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -125,20 +126,25 @@
 	 * */
 	private boolean failUnfulfillableRequest = true;
 
+	@Nullable
+	private final WorkerResourceSpec workerResourceSpec;
+
 	public SlotManagerImpl(
 			SlotMatchingStrategy slotMatchingStrategy,
 			ScheduledExecutor scheduledExecutor,
 			Time taskManagerRequestTimeout,
 			Time slotRequestTimeout,
 			Time taskManagerTimeout,
-			boolean waitResultConsumedBeforeRelease) {
+			boolean waitResultConsumedBeforeRelease,
+			@Nullable WorkerResourceSpec workerResourceSpec) {
 
 		this.slotMatchingStrategy = Preconditions.checkNotNull(slotMatchingStrategy);
 		this.scheduledExecutor = Preconditions.checkNotNull(scheduledExecutor);
 		this.taskManagerRequestTimeout = Preconditions.checkNotNull(taskManagerRequestTimeout);
 		this.slotRequestTimeout = Preconditions.checkNotNull(slotRequestTimeout);
 		this.taskManagerTimeout = Preconditions.checkNotNull(taskManagerTimeout);
 		this.waitResultConsumedBeforeRelease = waitResultConsumedBeforeRelease;
+		this.workerResourceSpec = workerResourceSpec;
 
 Review comment:
   We could pass resource-related configs and construct the default workerResourceSpec here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388723786
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -125,20 +126,25 @@
 	 * */
 	private boolean failUnfulfillableRequest = true;
 
+	@Nullable
+	private final WorkerResourceSpec workerResourceSpec;
+
 	public SlotManagerImpl(
 			SlotMatchingStrategy slotMatchingStrategy,
 			ScheduledExecutor scheduledExecutor,
 			Time taskManagerRequestTimeout,
 			Time slotRequestTimeout,
 			Time taskManagerTimeout,
-			boolean waitResultConsumedBeforeRelease) {
+			boolean waitResultConsumedBeforeRelease,
+			@Nullable WorkerResourceSpec workerResourceSpec) {
 
 		this.slotMatchingStrategy = Preconditions.checkNotNull(slotMatchingStrategy);
 		this.scheduledExecutor = Preconditions.checkNotNull(scheduledExecutor);
 		this.taskManagerRequestTimeout = Preconditions.checkNotNull(taskManagerRequestTimeout);
 		this.slotRequestTimeout = Preconditions.checkNotNull(slotRequestTimeout);
 		this.taskManagerTimeout = Preconditions.checkNotNull(taskManagerTimeout);
 		this.waitResultConsumedBeforeRelease = waitResultConsumedBeforeRelease;
+		this.workerResourceSpec = workerResourceSpec;
 
 Review comment:
   See my reply above.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403789023
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   It calls to the abstract method `getDefaultCpus`, which is currently overridden differently by different `ActiveResourceManagerFactory` implementations.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388867002
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   @xintongsong  So where is a document mentions what's the different among `slotResourceProfile`, `resourceProfile` and  `workerResourceSpec`. I can see there are some constraint among them but it is hard to understand from my side.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389432171
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
+ */
+public class WorkerResourceSpec {
 
 Review comment:
   Thanks for your explanation. Make sense to me.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408965071
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
 ##########
 @@ -524,10 +524,9 @@ private FinalApplicationStatus getYarnStatus(ApplicationStatus status) {
 	 * Request new container if pending containers cannot satisfy pending slot requests.
 	 */
 	private void requestYarnContainerIfRequired() {
-		int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		int pendingTaskManagerSlots = numPendingContainerRequests * numSlotsPerTaskManager;
+		int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		while (requiredTaskManagers-- > numPendingContainerRequests) {
 
 Review comment:
   ```suggestion
   		while (requiredTaskManagers > numPendingContainerRequests) {
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403035761
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1054,10 +1054,10 @@ protected abstract void internalDeregisterApplication(
 	 * Allocates a resource using the resource profile.
 	 *
 	 * @param resourceProfile The resource description
-	 * @return Collection of {@link ResourceProfile} describing the launched slots
+	 * @return whether the resource can be allocated
 	 */
 	@VisibleForTesting
-	public abstract Collection<ResourceProfile> startNewWorker(ResourceProfile resourceProfile);
+	public abstract boolean startNewWorker(ResourceProfile resourceProfile);
 
 Review comment:
   What happens if the RM can only start a multiple of `resourceProfile`? Wouldn't we calculate the number of available slots wrongly in this case?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389331707
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   Good point.
   I think it might be better to document it in `SlotManagerImpl` rather than `WorkerResourceSpec`. WDYT?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388723609
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   I tend to disagree, for the following reasons.
   - The default `WorkerResourceSpec` is needed for both static and dynamic slot allocation. For dynamic allocation, we need it for scenarios where some jobs do not provide fine grained resource requirement.
   - We need the default `WorkerResourceSpec` only for active RMs, but not for standalone RM. SM does not know which RM it runs on.
   - We need the default CPU cores for creating the default `WorkerResourceSpec`, which is depending on different configuration options per active RM. SM does not have that information.
   - According to the dependency injection principle, we should try to avoid creating dependencies inside the component if possible.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 6d44eb6678840e0bd288748962b33ee8539b3d5a Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/159265966) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210) 
   * ccdc15f99fc12ed13c1949a36af33ef2d5d806e3 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159283988) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219) 
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 09de8cdeb4d5f82649175e36fb95024f6cf55614 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159244566) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193) 
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 6d44eb6678840e0bd288748962b33ee8539b3d5a Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159265966) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210) 
   * ccdc15f99fc12ed13c1949a36af33ef2d5d806e3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160592479",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 80a7fd2aeed32cc6641e5830f28637d5761b7670 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160552877) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600) 
   * 7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160592479) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160592479",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 80a7fd2aeed32cc6641e5830f28637d5761b7670 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160552877) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600) 
   * 7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160592479) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-610253768
 
 
   I think you are right that the standalone `Dispatcher` should not fail if it cannot create the `WorkerResourceSpec` because there are some conflicting entries.
   
   Ok, then let's go with some arbitrary/default worker spec for the standalone case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r405329343
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1054,10 +1054,10 @@ protected abstract void internalDeregisterApplication(
 	 * Allocates a resource using the resource profile.
 	 *
 	 * @param resourceProfile The resource description
-	 * @return Collection of {@link ResourceProfile} describing the launched slots
+	 * @return whether the resource can be allocated
 	 */
 	@VisibleForTesting
-	public abstract Collection<ResourceProfile> startNewWorker(ResourceProfile resourceProfile);
+	public abstract boolean startNewWorker(ResourceProfile resourceProfile);
 
 Review comment:
   Looking more into this, I'm not sure whether returning the container size is the right approach.
   
   The problem is, the container resource contains two parts that are decided by different components.
   - Those contained in `WorkerResourceSpec` are decided by `SlotManager`: cpu, task heap, task off-heap, network, managed.
   - Those contained in `TaskExecutorProcessSpec` but not in `WorkerResourceSpec` are decided by `ResourceManager`: framework heap, framework off-heap, jvm metaspace, jvm overhead.
   
   If we return the size of the entire started container to SM, it does not help SM to decide whether there are resources wasted, because SM is not aware of how many resources in addition to `WorkerResourceSpec` that the framework and jvm need.
   
   An alternative is to exclude the framework and jvm resources and the remaining to SM. The problem is that, the excluded part is not constant and might be dependent on `WorkerResourceSpec`. To be specific, jvm overhead might be configured as a fraction of total process memory size. That means increasing the total size of `WorkerResourceSpec` may also lead to the increasing of jvm overhead.
   
   I think what should be returned by this API really depends on how we want to solve the resource wasting issue. TBH, I'm not sure when would we have a `SlotManager` implementation that deals with the Yarn resource wasting problem. I would suggest to create a follow-up ticket and leave it to when we do have such a SM implementation. WDYT?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160592479",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160592479) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403054549
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -207,6 +208,18 @@ public int getNumberFreeSlotsOf(InstanceID instanceId) {
 	}
 
 	@Override
+	public Map<WorkerResourceSpec, Integer> getPendingWorkerNums() {
+		final int pendingWorkerNum = (int) Math.ceil((double) pendingSlots.size() / numSlotsPerWorker);
+		if (pendingWorkerNum > 0) {
+			return Collections.singletonMap(
+				Preconditions.checkNotNull(defaultWorkerResourceSpec,
+					"There should never be pending slots/workers in standalone mode."),
 
 Review comment:
   I think this should not be necessary.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403060510
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1051,13 +1050,13 @@ protected abstract void internalDeregisterApplication(
 		@Nullable String optionalDiagnostics) throws ResourceManagerException;
 
 	/**
-	 * Allocates a resource using the resource profile.
+	 * Allocates a resource using the worker resource specification.
 	 *
-	 * @param resourceProfile The resource description
+	 * @param workerResourceSpec that describes the to be allocated worker.
 
 Review comment:
   ```suggestion
   	 * @param workerResourceSpec that describes the to be allocated worker
   ```
   
   param, return or throws tags are usually not terminated by a period.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388784879
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
+ */
+public class WorkerResourceSpec {
 
 Review comment:
   Maybe i missed some context, what's the difference between `WorkerResourceSpec` and `TaskExecutorResourceSpec`. They are very similar and share the same fields. Is it possible they will be different in the future?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 752470d53f899c7d415ad90e388c765563dd8fae Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160540563) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593) 
   * 80a7fd2aeed32cc6641e5830f28637d5761b7670 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160552877) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600) 
   * 7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f7068628ed9f11c4a4d0675230df530ceed251c5 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155768940) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 9528d6ec0d614bf1d66c1908f7679086f0fca961 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160485130) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550) 
   * 193ccd04ff0c811157bf073f778b5c30b3d80e3d Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160507168) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 0e50d9fbea8542a38840bac1150f527f343e5887 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160526443) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584) 
   * 752470d53f899c7d415ad90e388c765563dd8fae Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160540563) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403039671
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -807,23 +811,32 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 		return false;
 	}
 
-	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile requestedSlotResourceProfile) {
+		if (defaultWorkerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
+			return Optional.empty();
+		}
 
-		if (requestedSlots.isEmpty()) {
+		if (!Preconditions.checkNotNull(defaultSlotResourceProfile,
+			"defaultSlotResourceProfile should be null iff taskExecutorProcessSpec is null, which means standalone mode.")
+			.isMatching(requestedSlotResourceProfile)) {
+			// requested resource profile is unfulfillable
 			return Optional.empty();
-		} else {
-			final Iterator<ResourceProfile> slotIterator = requestedSlots.iterator();
-			final PendingTaskManagerSlot pendingTaskManagerSlot = new PendingTaskManagerSlot(slotIterator.next());
-			pendingSlots.put(pendingTaskManagerSlot.getTaskManagerSlotId(), pendingTaskManagerSlot);
+		}
 
 Review comment:
   Shouldn't we instead of returning an `Optional.empty()` fail with an invalid state exception because the current `SlotManagerImpl` does not support custom resource profiles yet?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388864118
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
+			LOG.info(
+				"The amount of cpu cores must be a positive integer on Yarn. Rounding {} up to the closest positive integer {}.",
+				cpuCoresDouble,
+				cpuCoresLong);
+		}
+		if (cpuCoresLong > Integer.MAX_VALUE) {
 
 Review comment:
   A precondition should work as defensive programming.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/152302512) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/152302512) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * ccdc15f99fc12ed13c1949a36af33ef2d5d806e3 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/159283988) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219) 
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389455647
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   I'm not sure why the `XxxConfiguration` is used at the first place. Probably there are some good reasons. From what I see, the `XxxConfiguration` clearly shows which information is needed for creating the `Xxx`.
   I think we can discuss the pros and cons of `XxxConfiguration` in another thread, maybe a code style discussion in ML? For this PR, I would also like to exclude it from the scope.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/152302512) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403051769
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
 ##########
 @@ -267,10 +267,9 @@ private void requestKubernetesPod() {
 	 * Request new pod if pending pods cannot satisfy pending slot requests.
 	 */
 	private void requestKubernetesPodIfRequired() {
-		final int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		final int pendingTaskManagerSlots = numPendingPodRequests * numSlotsPerTaskManager;
+		final int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		if (requiredTaskManagers > numPendingPodRequests) {
 
 Review comment:
   Unrelated. Wouldn't a while loop be better here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388870561
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
 ##########
 @@ -272,10 +267,9 @@ private void requestKubernetesPod() {
 	 * Request new pod if pending pods cannot satisfy pending slot requests.
 	 */
 	private void requestKubernetesPodIfRequired() {
-		final int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		final int pendingTaskManagerSlots = numPendingPodRequests * numSlotsPerTaskManager;
+		final int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		if (requiredTaskManagers > numPendingPodRequests) {
 
 Review comment:
   `startNewWorker` is a public interface, and its return value is checked by `SlotManager` for creating pending slots.
   For `requestKubernetesPodIfRequired`, I don't see how its return value is needed, at least for now. And since it's a internal private method, I think we can always add that return value if needed later.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 752470d53f899c7d415ad90e388c765563dd8fae Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160540563) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593) 
   * 80a7fd2aeed32cc6641e5830f28637d5761b7670 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160552877) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403799016
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1206,8 +1206,12 @@ public Void retrievePayload(ResourceID resourceID) {
 	//  Resource Management
 	// ------------------------------------------------------------------------
 
-	protected int getNumberRequiredTaskManagerSlots() {
-		return slotManager.getNumberPendingTaskManagerSlots();
+	protected int getNumberRequiredTaskManagers() {
+		return getPendingWorkerNums().values().stream().reduce(0, Integer::sum);
 
 Review comment:
   IIUC, this is because the RM implementations are ignoring the `WorkerResourceSpec` passed to `startNewWorker`? Then would it be better to add this check in `startNewWorker`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388723699
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerConfiguration.java
 ##########
 @@ -43,19 +46,23 @@
 	private final Time taskManagerTimeout;
 	private final boolean waitResultConsumedBeforeRelease;
 	private final boolean evenlySpreadOutSlots;
+	@Nullable
+	private final WorkerResourceSpec defaultWorkerResourceSpec;
 
 	public SlotManagerConfiguration(
 			Time taskManagerRequestTimeout,
 			Time slotRequestTimeout,
 			Time taskManagerTimeout,
 			boolean waitResultConsumedBeforeRelease,
-			boolean evenlySpreadOutSlots) {
+			boolean evenlySpreadOutSlots,
+			@Nullable WorkerResourceSpec defaultWorkerResourceSpec) {
 
 Review comment:
   The default WorkerResourceSpec is needed for both static and dynamic slot allocation. For dynamic allocation, we need it for scenarios where some jobs do not provide fine grained resource requirement.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f7068628ed9f11c4a4d0675230df530ceed251c5 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155768940) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389333966
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   True, we have the `@Nullable WorkerResourceSpec` along the code path for creating the `SlotManagerImpl`. To be specific, in the following classes.
   - `ResourceManagerRuntimeServiceConfiguration`
   - `SlotManagerConfiguration`
   - `SlotManagerImpl`
   - `SlotManagerBuilder` (test-only)
   
   I guess the root cause is that we decide `WorkerResourceSpec` differently (whether `null` and the `cpu` config options) across different RMs, so we have to pass it from the `ResourceManagerFactory`. Unfortunately, ATM I don't see a good way to resolve that. Do you have any suggestion on that?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 9528d6ec0d614bf1d66c1908f7679086f0fca961 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160485130) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389439416
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   There are too many `XxxConfiguration` which make the call chain much longer. We might 
    refactor to reduce some poor `XxxConfiguration` later. But it doesn't have to be included in this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-614541196
 
 
   Rebased onto the latest master, and added fix for PerJobMiniClusterFactoryTest.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389440058
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   Or in this pr you can try to directly pass `WorkerResourceSpec` to service's  & manager's `fromConfiguration` instead of configuration's & build's.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 09de8cdeb4d5f82649175e36fb95024f6cf55614 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159244566) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193) 
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408960456
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1206,8 +1206,12 @@ public Void retrievePayload(ResourceID resourceID) {
 	//  Resource Management
 	// ------------------------------------------------------------------------
 
-	protected int getNumberRequiredTaskManagerSlots() {
-		return slotManager.getNumberPendingTaskManagerSlots();
+	protected int getNumberRequiredTaskManagers() {
+		return getPendingWorkerNums().values().stream().reduce(0, Integer::sum);
+	}
+
+	protected Map<WorkerResourceSpec, Integer> getPendingWorkerNums() {
+		return slotManager.getPendingWorkerNums();
 
 Review comment:
   Should we rename this method into `getRequiredResources` or so?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * ccdc15f99fc12ed13c1949a36af33ef2d5d806e3 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/159283988) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219) 
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403062228
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/ResourceActions.java
 ##########
 @@ -37,12 +37,12 @@
 	void releaseResource(InstanceID instanceId, Exception cause);
 
 	/**
-	 * Requests to allocate a resource with the given {@link ResourceProfile}.
+	 * Requests to allocate a resource with the given {@link WorkerResourceSpec}.
 	 *
-	 * @param resourceProfile for the to be allocated resource
+	 * @param workerResourceSpec for the to be allocated worker
 
 Review comment:
   Usually `@param` requires a noun phrase: `@param workerResourceSpec workerResourceSpec specifies the size of the to be allocated resource`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388691152
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesResourceManagerFactory.java
 ##########
 @@ -80,4 +84,10 @@ public static KubernetesResourceManagerFactory getInstance() {
 			fatalErrorHandler,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(Configuration configuration) {
 
 Review comment:
   This duplicates with `KubernetesResourceManager#getCpuCores`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408959505
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
 ##########
 @@ -267,10 +267,9 @@ private void requestKubernetesPod() {
 	 * Request new pod if pending pods cannot satisfy pending slot requests.
 	 */
 	private void requestKubernetesPodIfRequired() {
-		final int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		final int pendingTaskManagerSlots = numPendingPodRequests * numSlotsPerTaskManager;
+		int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		while (requiredTaskManagers-- > numPendingPodRequests) {
 
 Review comment:
   ```suggestion
   		while (requiredTaskManagers > numPendingPodRequests) {
   ```
   I think `requestKubernetesPod` will increase `numPendingPodRequests`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408964926
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
 ##########
 @@ -524,10 +524,9 @@ private FinalApplicationStatus getYarnStatus(ApplicationStatus status) {
 	 * Request new container if pending containers cannot satisfy pending slot requests.
 	 */
 	private void requestYarnContainerIfRequired() {
-		int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		int pendingTaskManagerSlots = numPendingContainerRequests * numSlotsPerTaskManager;
+		int requiredTaskManagers = getNumberRequiredTaskManagers();
 
 Review comment:
   ```suggestion
   		final int requiredTaskManagers = getNumberRequiredTaskManagers();
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-614635583
 
 
   Thanks a lot, Till~

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388865070
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   The problem is not about calling `StandaloneResourceManager#startNewWorker`. It is about not having a `WorkerResourceSpec` on standalone setups.
   
   For a standalone cluster, use may have TM resource configurations only on the TM machines. That means trying to create a `WorkerResourceSpec` from the configuration on JM might lead to failures due to missing of memory configurations. I think it does not make sense to force the users to set TM configurations on the JM machine, because they are not really used.
   
   We have run into such problems during the 1.9 release testing, where a standalone JM failed because it tries to calculate a default TM managed memory size.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388865522
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   It should be able to merge into `if (!resourceActions.allocateResource(workerResourceSpec))` branch. But I am not very sure because the next condition is
   
   ```
   if (!Preconditions.checkNotNull(slotResourceProfile,
   			"slotResourceProfile should be null iff workerResourceSpec is null, which means standalone mode.")
   			.isMatching(resourceProfile))
   ```
   
   where we now have three profile(including workerResourceSpec) things become quite complex.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595612104
 
 
   Thanks for the review, @KarmaGYZ.
   Please see my explanations in the replies to your comments.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-609556553

Thanks for the review, @tillrohrmann.

Regarding the nullable `defaultWorkerResourceSpec` and `defaultSlotResourceProfile`, I'm trying to understand and summarize your concerns as follows. Please correct me if I'm wrong.
- It is not explicit what the null values means and should be handled.
- It makes `SlotManagerImpl` kind of inferring which deployment (standalone or active) it works on, and behave differently.
These concerns make good sense to me.

Actually, I have also considered having a special `WorkerResourceSpec.UNKNOWN`. My concern is that, it may require all the places that uses `WorkerResourceSpec` to check and handle the special value, which seems unnecessary to me. E.g., for `ResourceActions#allocateResource`, the argument `WorkerResourceProfile` should never be `UNKNOWN` because SM should always request workers from RM with specific worker resource spec. If we add `WorkerResourceSpec.UNKNOWN`, we have to either carefully make sure it is never passed to `allocateResource` in all the implementations of `SlotManager`, or check argument in all implementations of `ResourceAction`.

I think `WorkerResourceSpec.UNKNOWN` is different from `ResourceProfile.UNKNOWN` or `ResourceSpec.UNKNOWN`. For `ResourceProfile`/`ResourceSpec`, `UNKNOWN` means we need some resource but we don't known how many exactly. For `defaultWorkerResourceSpec`, the semantic we actually need for standalone clusters is "the default worker does not exist", or in other words "by default we do not request workers".

Combining your concerns and mine, and the above understanding, I would propose the following approach.
- Introduce an `DefaultWorkerSlotResourceProvider`, with methods `Optional<WorkerResourceSpec> getDefaultWorkerResourceSpec` and `Optional<ResourceProfile> getDefaultSlotResourceProfile`.
- For standalone clusters, we can pass a special `DefaultWorkerSlotResourceProvider.NO_DEFAULTS` to `SlotManagerImpl`, which returns `Optional.empty()` for both methods. The absent optional value clearly reflects the semantics of "not existing".
- `SlotManagerImpl` no longer needs to assume which RM implementation it works with. It simply not allocate resource from RM if `DefaultWorkerSlotResourceProvider#getDefaultWorkerResourceSpec` returns empty.

WDYT?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388861967
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
+			LOG.info(
+				"The amount of cpu cores must be a positive integer on Yarn. Rounding {} up to the closest positive integer {}.",
+				cpuCoresDouble,
+				cpuCoresLong);
+		}
+		if (cpuCoresLong > Integer.MAX_VALUE) {
 
 Review comment:
   I think it is not robust to make the correctness of `YarnResourceManagerFactory` depending on another class `YarnClusterDescriptor`. There is no harm for double checking it, right?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160592479",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160592479) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * df11b25dc9094509f24cec5484a6c5b01622d3f8 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155803652) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-611279664
 
 
   @tillrohrmann 
   I've addressed your comments, except for one about returning size of the started container to slot manager. Please see the corresponding comment thread for details.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389332200
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   I tend to think it differently.
   I think the semantic of a `WorkerResourceSpec` being `null` belongs to the context where it is used, not the class itself. E.g., when we create the `SlotManagerImpl`, the `WorkerResourceSpec` means it is on a standalone setup and SM should not request any worker from RM. On the other hand, when we create a `TaskExecutorProcessSpec` from the `WorkerResourceSpce`, it must not be `null`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389434358
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
+			LOG.info(
+				"The amount of cpu cores must be a positive integer on Yarn. Rounding {} up to the closest positive integer {}.",
+				cpuCoresDouble,
+				cpuCoresLong);
+		}
+		if (cpuCoresLong > Integer.MAX_VALUE) {
 
 Review comment:
   I am not against with the check. I just feel the `Integer.MAX_VALUE` checking will not take too much effect. Since the `YarnMaxVcores` is far less than `Integer.MAX_VALUE`.
   
   Centainly, you could keep the current check. It is safer when converting `long` to `int`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r404080033
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
+		final TaskExecutorProcessSpec taskExecutorProcessSpec = TaskExecutorProcessUtils
+			.newProcessSpecBuilder(configuration)
+			.withCpuCores(getDefaultCpus(configuration))
+			.build();
+		return WorkerResourceSpec.fromTaskExecutorProcessSpec(taskExecutorProcessSpec);
+	}
+
+	protected abstract CPUResource getDefaultCpus(Configuration configuration);
 
 Review comment:
   My suggestion would be to pass it to the `ActiveResourceManagerFactory` when creating it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403823418
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
+		final TaskExecutorProcessSpec taskExecutorProcessSpec = TaskExecutorProcessUtils
+			.newProcessSpecBuilder(configuration)
+			.withCpuCores(getDefaultCpus(configuration))
+			.build();
+		return WorkerResourceSpec.fromTaskExecutorProcessSpec(taskExecutorProcessSpec);
+	}
+
+	protected abstract CPUResource getDefaultCpus(Configuration configuration);
 
 Review comment:
   If the deriving of cpu cores and default worker spec is too detailed for `ActiveResourceManager`, I would suggest to add an abstract method `getDefaultWorkerSpecFactory` to return different `WorkerResourceSpecFactory` for different `ActiveResourceManager` implementations.
   
   My concern for passing `WorkerResourceSpecFactory` or `WorkerResourceSpec` to `ActiveResourceManager` is that, neither the constructor nor the `createResourceManager` method seems to be a good place to pass them in IMO.
   - For the constructor, `ActiveResourceManager` is abstract and its implementations are either singleton (K8s/Yarn) or created by cluster entry point (Mesos). I think it is too detail for the cluster entry point to pass in different `WorkerResourceSpecFactory`/`WorkerResourceSpec` to the resource manager factory, not to mention breaking singleton properties on K8s/Yarn.
   - For `createResourceManager`, it is called by `DefaultDispatcherResourceManagerComponentFactory`, which is not aware of the different deployments either.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388721739
 
 

 ##########
 File path: flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
 ##########
 @@ -98,4 +101,10 @@ public MesosResourceManagerFactory(@Nonnull MesosServices mesosServices, @Nonnul
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(Configuration configuration) {
 
 Review comment:
   Same here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/151939995) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r404630795
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1054,10 +1054,10 @@ protected abstract void internalDeregisterApplication(
 	 * Allocates a resource using the resource profile.
 	 *
 	 * @param resourceProfile The resource description
-	 * @return Collection of {@link ResourceProfile} describing the launched slots
+	 * @return whether the resource can be allocated
 	 */
 	@VisibleForTesting
-	public abstract Collection<ResourceProfile> startNewWorker(ResourceProfile resourceProfile);
+	public abstract boolean startNewWorker(ResourceProfile resourceProfile);
 
 Review comment:
   Hmm, but with this API we will waste resources because the caller thinks it has started a worker of the specified size, right? An idea how to improve the situation is to let the `startNewWorker` method return the size of the container it has started. This would allow the caller to react to it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389439416
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   There are too many `XxxConfiguration` which make the call chain much longer. We might refactor to reduce some poor `XxxConfiguration` later. But it doesn't have to be included in this PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403056960
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
 ##########
 @@ -156,8 +156,8 @@ protected void internalDeregisterApplication(ApplicationStatus finalStatus, @Nul
 	}
 
 	@Override
-	public boolean startNewWorker(ResourceProfile resourceProfile) {
-		LOG.info("Starting new worker with resource profile, {}", resourceProfile);
+	public boolean startNewWorker(WorkerResourceSpec workerResourceSpec) {
+		LOG.info("Starting new worker with worker resource spec, {}", workerResourceSpec);
 		requestKubernetesPod();
 
 Review comment:
   I guess that we need a check that the configured pod fulfills `workerResourceSpec`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388696487
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerConfiguration.java
 ##########
 @@ -43,19 +46,23 @@
 	private final Time taskManagerTimeout;
 	private final boolean waitResultConsumedBeforeRelease;
 	private final boolean evenlySpreadOutSlots;
+	@Nullable
+	private final WorkerResourceSpec defaultWorkerResourceSpec;
 
 	public SlotManagerConfiguration(
 			Time taskManagerRequestTimeout,
 			Time slotRequestTimeout,
 			Time taskManagerTimeout,
 			boolean waitResultConsumedBeforeRelease,
-			boolean evenlySpreadOutSlots) {
+			boolean evenlySpreadOutSlots,
+			@Nullable WorkerResourceSpec defaultWorkerResourceSpec) {
 
 Review comment:
   Same as above. Not sure the `defaultWorkerResourceSpec` is a common field for both static and dynamic SM.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-614413676
 
 
   Thanks @tillrohrmann. Comments addressed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r401738574
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -125,20 +126,29 @@
 	 * */
 	private boolean failUnfulfillableRequest = true;
 
+	/**
+	 * The default resource spec of workers to request. It could be null iff on standalone/local setups, in which cases
+	 * the {@link SlotManagerImpl} will never request any new worker.
+	 */
+	@Nullable
+	private final WorkerResourceSpec defaultWorkerResourceSpec;
 
 Review comment:
   A nice side-effect would be that everywhere where we now pass `null` we would have a nice and descriptive name `WorkerResourceSpec.UNSPECIFIED` or `WorkerResourceSpec.getUnknownSpec()`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f7068628ed9f11c4a4d0675230df530ceed251c5 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155768940) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733) 
   * df11b25dc9094509f24cec5484a6c5b01622d3f8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155803652) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403805525
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -807,23 +811,32 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 		return false;
 	}
 
-	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile requestedSlotResourceProfile) {
+		if (defaultWorkerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
+			return Optional.empty();
+		}
 
-		if (requestedSlots.isEmpty()) {
+		if (!Preconditions.checkNotNull(defaultSlotResourceProfile,
+			"defaultSlotResourceProfile should be null iff taskExecutorProcessSpec is null, which means standalone mode.")
+			.isMatching(requestedSlotResourceProfile)) {
+			// requested resource profile is unfulfillable
 			return Optional.empty();
-		} else {
-			final Iterator<ResourceProfile> slotIterator = requestedSlots.iterator();
-			final PendingTaskManagerSlot pendingTaskManagerSlot = new PendingTaskManagerSlot(slotIterator.next());
-			pendingSlots.put(pendingTaskManagerSlot.getTaskManagerSlotId(), pendingTaskManagerSlot);
+		}
 
 Review comment:
   I'm not sure whether this should be an invalid state exception. 
   
   It is true that all the slot requests should have unknown resource profile ATM. However, I think `SlotManagerImpl` does not necessarily need to know whether there're custom profiles or not. As long as the requested resource profile can be matched by the default slot profile, it should be able to allocate the slot. And if the requested slot profile is larger than the default slot, an `UnfulfillableSlotRequestException` should be thrown.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388861404
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
+ */
+public class WorkerResourceSpec {
 
 Review comment:
   There was a discussion about whether reusing `TaskExecutorResourceSpec` in the design google doc. We have decided to not reuse it, and I'll quote what Till commented on the doc here.
   > TaskExecutorResourceSpec defines for me the resources of a TaskExecutor. The SlotManager on the other hand is only interested in a certain set of resources which happen to be the exact same set as TaskExecutorResourceSpec offers. However, the latter might evolve adding more resource specifications. Hence, by reusing TaskExecutorResourceSpec we would also couple the SlotManager to the TaskExecutor's resource specification which does not necessarily need to be the same.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403055120
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
 ##########
 @@ -524,10 +524,9 @@ private FinalApplicationStatus getYarnStatus(ApplicationStatus status) {
 	 * Request new container if pending containers cannot satisfy pending slot requests.
 	 */
 	private void requestYarnContainerIfRequired() {
-		int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		int pendingTaskManagerSlots = numPendingContainerRequests * numSlotsPerTaskManager;
+		int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		if (requiredTaskManagers > numPendingContainerRequests) {
 
 Review comment:
   Same here with the `while` loop. But also unrelated to this PR as the other comment.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 193ccd04ff0c811157bf073f778b5c30b3d80e3d Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160507168) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571) 
   * 0e50d9fbea8542a38840bac1150f527f343e5887 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160526443) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584) 
   * 752470d53f899c7d415ad90e388c765563dd8fae UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r404084546
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/ResourceActions.java
 ##########
 @@ -42,9 +40,9 @@
 	 * Requests to allocate a resource with the given {@link ResourceProfile}.
 	 *
 	 * @param resourceProfile for the to be allocated resource
-	 * @return Collection of {@link ResourceProfile} describing the allocated slots
+	 * @return whether the resource can be allocated
 	 */
-	Collection<ResourceProfile> allocateResource(ResourceProfile resourceProfile);
+	boolean allocateResource(ResourceProfile resourceProfile);
 
 Review comment:
   All right, understood.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r404083978
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -807,23 +811,32 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 		return false;
 	}
 
-	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile requestedSlotResourceProfile) {
+		if (defaultWorkerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
+			return Optional.empty();
+		}
 
-		if (requestedSlots.isEmpty()) {
+		if (!Preconditions.checkNotNull(defaultSlotResourceProfile,
+			"defaultSlotResourceProfile should be null iff taskExecutorProcessSpec is null, which means standalone mode.")
+			.isMatching(requestedSlotResourceProfile)) {
+			// requested resource profile is unfulfillable
 			return Optional.empty();
-		} else {
-			final Iterator<ResourceProfile> slotIterator = requestedSlots.iterator();
-			final PendingTaskManagerSlot pendingTaskManagerSlot = new PendingTaskManagerSlot(slotIterator.next());
-			pendingSlots.put(pendingTaskManagerSlot.getTaskManagerSlotId(), pendingTaskManagerSlot);
+		}
 
 Review comment:
   Failing in case that the requested slot profile is too large sound good to me.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388697567
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -125,20 +126,25 @@
 	 * */
 	private boolean failUnfulfillableRequest = true;
 
+	@Nullable
+	private final WorkerResourceSpec workerResourceSpec;
+
 	public SlotManagerImpl(
 			SlotMatchingStrategy slotMatchingStrategy,
 			ScheduledExecutor scheduledExecutor,
 			Time taskManagerRequestTimeout,
 			Time slotRequestTimeout,
 			Time taskManagerTimeout,
-			boolean waitResultConsumedBeforeRelease) {
+			boolean waitResultConsumedBeforeRelease,
+			@Nullable WorkerResourceSpec workerResourceSpec) {
 
 		this.slotMatchingStrategy = Preconditions.checkNotNull(slotMatchingStrategy);
 		this.scheduledExecutor = Preconditions.checkNotNull(scheduledExecutor);
 		this.taskManagerRequestTimeout = Preconditions.checkNotNull(taskManagerRequestTimeout);
 		this.slotRequestTimeout = Preconditions.checkNotNull(slotRequestTimeout);
 		this.taskManagerTimeout = Preconditions.checkNotNull(taskManagerTimeout);
 		this.waitResultConsumedBeforeRelease = waitResultConsumedBeforeRelease;
+		this.workerResourceSpec = workerResourceSpec;
 
 Review comment:
   Also, there is no need to judge if the environment is Standalone or not. `resourceActions.allocateResource` will return false in such scenario.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388691635
 
 

 ##########
 File path: flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosResourceManagerFactory.java
 ##########
 @@ -98,4 +101,10 @@ public MesosResourceManagerFactory(@Nonnull MesosServices mesosServices, @Nonnul
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(Configuration configuration) {
 
 Review comment:
   This duplicates with `MesosTaskManagerParameters#getCpuCores`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151939995) 
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 9528d6ec0d614bf1d66c1908f7679086f0fca961 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160485130) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388693078
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   If I understand it correctly, SM, instead of RM,  should take care of the `WorkerResourceSpec`. Meanwhile, only static SM needs a default `WorkerResourceSpec`. If that is the case, I prefer to move this logic inside the current `SlotManagerImpl`. This may also prevent getting the default CPU cores in ResourceManager.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151939995) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388862718
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
 
 Review comment:
   True, we have not exposed `taskmanager.cpu.cores`. But we might in the future, and if we do, it could be easily overlooked to add this check later. Again, I believe there's no harm for over checking it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-609814865
 
 
   I see your point @xintongsong and agree that the caller (`SlotManager`) should not call the `RM` with some invalid value. I guess what we would really need is an `ActiveSlotManager` and a `SlotManager` which cannot and won't try to allocate new resources.
   
   Why is it that the standalone `SlotManager` cannot generate a `WorkerResourceSpec`? Shouldn't we be able to calculate it? Given that the actual value should not matter, we could also default to some random/arbitrary value.
   
   If this is not possible, then I'm ok with adding a `DefaultWorkerSlotResourceProvider` and having a `DefaultWorkerSlotResourceProvider.NO_RESOURCES` or so.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 0e50d9fbea8542a38840bac1150f527f343e5887 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160526443) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584) 
   * 752470d53f899c7d415ad90e388c765563dd8fae Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160540563) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593) 
   * 80a7fd2aeed32cc6641e5830f28637d5761b7670 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388823107
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
+			LOG.info(
+				"The amount of cpu cores must be a positive integer on Yarn. Rounding {} up to the closest positive integer {}.",
+				cpuCoresDouble,
+				cpuCoresLong);
+		}
+		if (cpuCoresLong > Integer.MAX_VALUE) {
 
 Review comment:
   I think it should not happen here. Because we have checked the configured vcores in `YarnClusterDescriptor`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403789023
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   It calls to the abstract method `getDefaultCpus`, which is currently overridden differently by different `ActiveResourceManagerFactory` implementations.
   
   Alternatively, we could make this a static method in either `ActiveResourceManagerFactory` of `TaskExecutorProcessUtils`, and make cpu cores an argument of this method. Just not sure how necessary it is.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403054142
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManager.java
 ##########
 @@ -51,7 +53,7 @@
 
 	int getNumberFreeSlotsOf(InstanceID instanceId);
 
-	int getNumberPendingTaskManagerSlots();
+	Map<WorkerResourceSpec, Integer> getPendingWorkerNums();
 
 Review comment:
   `JavaDoc` missing.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r404085816
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1206,8 +1206,12 @@ public Void retrievePayload(ResourceID resourceID) {
 	//  Resource Management
 	// ------------------------------------------------------------------------
 
-	protected int getNumberRequiredTaskManagerSlots() {
-		return slotManager.getNumberPendingTaskManagerSlots();
+	protected int getNumberRequiredTaskManagers() {
+		return getPendingWorkerNums().values().stream().reduce(0, Integer::sum);
 
 Review comment:
   If this is the entry point, then we can do the check there. The important bit is to maintain the invariant that we do not start containers for `WorkerResourceSpecs` which are larger than the container.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403793315
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1054,10 +1054,10 @@ protected abstract void internalDeregisterApplication(
 	 * Allocates a resource using the resource profile.
 	 *
 	 * @param resourceProfile The resource description
-	 * @return Collection of {@link ResourceProfile} describing the launched slots
+	 * @return whether the resource can be allocated
 	 */
 	@VisibleForTesting
-	public abstract Collection<ResourceProfile> startNewWorker(ResourceProfile resourceProfile);
+	public abstract boolean startNewWorker(ResourceProfile resourceProfile);
 
 Review comment:
   In this commit, we have moved the logics of calculating number of new pending slots into `SlotManagerImpl`. Therefore, `SlotManagerImpl` only needs to know whether the new worker is started or not, and should be able to correctly calculate the number of available slots.
   
   In the final state, RM accepts `WorkerResourceSpec` instead of slot `ResourceProfile` for starting new workers. It would become a SM decision that how many multiple of `ResourceProfile` the worker should be start with. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403063198
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
 ##########
 @@ -288,7 +288,7 @@ protected void internalDeregisterApplication(
 	}
 
 	@Override
-	public boolean startNewWorker(ResourceProfile resourceProfile) {
+	public boolean startNewWorker(WorkerResourceSpec workerResourceSpec) {
 
 Review comment:
   I think we should ensure that `workerResourceSpec` can be fulfilled by the actual container size the `YarnResourceManager` starts.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403028625
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   I'm wondering whether this should really be a method belonging to the `ActiveResourceManagerFactory`. If I look at the body, then it could perfectly be a static method. No need to be method of `ActiveResourceManagerFactory` since it does not access any state.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r401737501
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -125,20 +126,29 @@
 	 * */
 	private boolean failUnfulfillableRequest = true;
 
+	/**
+	 * The default resource spec of workers to request. It could be null iff on standalone/local setups, in which cases
+	 * the {@link SlotManagerImpl} will never request any new worker.
+	 */
+	@Nullable
+	private final WorkerResourceSpec defaultWorkerResourceSpec;
 
 Review comment:
   Instead of making this field nullable, I think it would be better to have an `UnspecifiedWorkerResourceSpec` instance which we can pass to the `SlotManagerImpl` in case it runs in standalone mode. I think this would also better reflect that the `SlotManagerImpl` does not really care whether it's running in a Standalone- or YarnResourceManager. At the moment, the `SlotManagerImpl` contains code which behaves differently depending on whether this field is null or not.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

Thanks for the review, @tillrohrmann.

Actually, I have also considered having a special `WorkerResourceSpec.UNKNOWN`. My concern is that, it may lead to all the places that uses `WorkerResourceSpec` need to check and handle the special value, which seems unnecessary to me. E.g., for `ResourceActions#allocateResource`, the argument `WorkerResourceProfile` should never be `UNKNOWN` because SM should always request workers from RM with specific worker resource spec. If we add `WorkerResourceSpec.UNKNOWN`, we have to either carefully make sure it is never passed to `allocateResource` in all the implementations of `SlotManager`, or check argument in all implementations of `ResourceAction`.

WDYT?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408972482
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1054,10 +1054,10 @@ protected abstract void internalDeregisterApplication(
 	 * Allocates a resource using the resource profile.
 	 *
 	 * @param resourceProfile The resource description
-	 * @return Collection of {@link ResourceProfile} describing the launched slots
+	 * @return whether the resource can be allocated
 	 */
 	@VisibleForTesting
-	public abstract Collection<ResourceProfile> startNewWorker(ResourceProfile resourceProfile);
+	public abstract boolean startNewWorker(ResourceProfile resourceProfile);
 
 Review comment:
   Sounds good to me. Thanks for the background information.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403053905
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1206,8 +1206,12 @@ public Void retrievePayload(ResourceID resourceID) {
 	//  Resource Management
 	// ------------------------------------------------------------------------
 
-	protected int getNumberRequiredTaskManagerSlots() {
-		return slotManager.getNumberPendingTaskManagerSlots();
+	protected int getNumberRequiredTaskManagers() {
+		return getPendingWorkerNums().values().stream().reduce(0, Integer::sum);
 
 Review comment:
   For the time being I would add a check state which ensures that the `WorkerResourceSpecs` can all be fulfilled by the container size we are using to start new containers/pods. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408960934
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManager.java
 ##########
 @@ -51,7 +53,12 @@
 
 	int getNumberFreeSlotsOf(InstanceID instanceId);
 
-	int getNumberPendingTaskManagerSlots();
+	/**
+	 * Get number of workers SlotManager requested from {@link ResourceActions} that are not yet fulfilled.
+	 * @return a map whose key set is all the unique resource specs of the pending workers,
+	 * and the corresponding value is number of pending workers of that resource spec.
+	 */
+	Map<WorkerResourceSpec, Integer> getPendingWorkerNums();
 
 Review comment:
   ```suggestion
   	Map<WorkerResourceSpec, Integer> getRequiredResources();
   ```
   maybe?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388824888
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
 
 Review comment:
   Do we have any chance they are not equal? Since we do not expose the `taskmanager.cpu.cores` to users now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403046204
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/ResourceActions.java
 ##########
 @@ -42,9 +40,9 @@
 	 * Requests to allocate a resource with the given {@link ResourceProfile}.
 	 *
 	 * @param resourceProfile for the to be allocated resource
-	 * @return Collection of {@link ResourceProfile} describing the allocated slots
+	 * @return whether the resource can be allocated
 	 */
-	Collection<ResourceProfile> allocateResource(ResourceProfile resourceProfile);
+	boolean allocateResource(ResourceProfile resourceProfile);
 
 Review comment:
   Wouldn't it be required for future developments that we can tell `allocateResource` how big resource-wise the worker should be?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-596297817
 
 
   @flinkbot run travis
   @flinkbot run azure

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389331955
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   My concern is that we have many `@Nullable` `WorkerResourceSpec` parameters. The semantic of `null` `WorkerResourceSpec` belongs to itself. So we can document the design in the class doc of `WorkerResourceSpec` and linked to this universe doc where a nullable param in use.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408962507
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -199,6 +200,14 @@ public int getNumberFreeSlotsOf(InstanceID instanceId) {
 	}
 
 	@Override
+	public Map<WorkerResourceSpec, Integer> getPendingWorkerNums() {
+		final int pendingWorkerNum = (int) Math.ceil((double) pendingSlots.size() / numSlotsPerWorker);
 
 Review comment:
   If we want to get rid of back and forth casting and double calculations:
   ```suggestion
   		final int pendingWorkerNum = (pendingSlots.size() - 1) / numSlotsPerWorker + 1;
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389440058
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   Or in this pr you can try to directly pass `WorkerResourceSpec` to service's  & manager's `fromConfiguration` instead of configuration's & builder's.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403037228
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -153,6 +155,8 @@ public SlotManagerImpl(
 		this.waitResultConsumedBeforeRelease = waitResultConsumedBeforeRelease;
 		this.defaultWorkerResourceSpec = defaultWorkerResourceSpec;
 		this.numSlotsPerWorker = numSlotsPerWorker;
+		this.defaultSlotResourceProfile = defaultWorkerResourceSpec != null ?
+			generateDefaultSlotResourceProfile(defaultWorkerResourceSpec, numSlotsPerWorker) : null;
 
 Review comment:
   Instead of setting `defaultSlotResourceProfile` to `null`, can't we set it to `ResourceProfile.UNKNOWN` or something like this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388878181
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   @TisonKun
   Please see my reply to Yang for why calling `allocateResource` on standalone setup. For the same reason, we cannot merge this into the `if (!resourceActions.allocateResource(workerResourceSpec))` branch.
   
   Regarding `slotResourceProfile`, `resourceProfile` and `workerResourceSpec`.
   - `slotResourceProfile` describes the resource capacity of a slot.
   - `resourceProfile` describes the resource requirement of a slot request.
   - `workerResourceSpec` describes the resource requirement of a worker (TM) to request.
   
   I think you're right, these things are indeed hard to understand. How do you think about renaming `resourceProfile` to `requestedSlotResourceProfile`, and `slotResourceProfile` to `defaultSlotResourceProfile`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388867002
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   @xintongsong  So where is a document mentions what's the different among `slotResourceProfile`, `resourceProfile` and  `workerResourceSpec`. I can see there are some constraints/dependencies among them but it is hard to understand from my side.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388721558
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
 
 Review comment:
   Yes, I'm aware of this.
   It's just we are currently having both default `TaskExecutorProcessSpec` and default `WorkerResourceSpec`, and will migrate from the former to the latter for the active RMs one by one. The `ActiveResourceManager#getCpuCores` could be removed soon after all active RMs are migrated. That's why I didn't bother to de-duplicate them.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r409483913
 
 

 ##########
 File path: flink-clients/src/main/java/org/apache/flink/client/program/PerJobMiniClusterFactory.java
 ##########
 @@ -100,8 +100,10 @@ private MiniClusterConfiguration getMiniClusterConfig(int maximumParallelism) {
 			ConfigConstants.LOCAL_NUMBER_TASK_MANAGER,
 			ConfigConstants.DEFAULT_LOCAL_NUMBER_TASK_MANAGER);
 
-		// we have to use the maximum parallelism as a default here, otherwise streaming pipelines would not run
-		int numSlotsPerTaskManager = configuration.getInteger(TaskManagerOptions.NUM_TASK_SLOTS, maximumParallelism);
+		int numSlotsPerTaskManager = configuration.getOptional(TaskManagerOptions.NUM_TASK_SLOTS)
+			.orElseGet(() -> maximumParallelism > 0 ?
+				(int) Math.ceil((double) maximumParallelism / numTaskManagers) :
 
 Review comment:
   Can't we use `MathUtils.divideRoundUp` here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389238604
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
+ */
+public class WorkerResourceSpec {
+	private final CPUResource cpuCores;
+
+	private final MemorySize taskHeapSize;
+
+	private final MemorySize taskOffHeapSize;
+
+	private final MemorySize networkMemSize;
+
+	private final MemorySize managedMemSize;
+
+	public WorkerResourceSpec(
+		CPUResource cpuCores,
+		MemorySize taskHeapSize,
+		MemorySize taskOffHeapSize,
+		MemorySize networkMemSize,
+		MemorySize managedMemSize) {
+
+		this.cpuCores = Preconditions.checkNotNull(cpuCores);
+		this.taskHeapSize = Preconditions.checkNotNull(taskHeapSize);
+		this.taskOffHeapSize = Preconditions.checkNotNull(taskOffHeapSize);
+		this.networkMemSize = Preconditions.checkNotNull(networkMemSize);
+		this.managedMemSize = Preconditions.checkNotNull(managedMemSize);
+	}
+
+	@VisibleForTesting
+	public WorkerResourceSpec(
 
 Review comment:
   Share we have a inner Builder to reduce multiple testing purpose constructor override?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r401726831
 
 

 ##########
 File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpecTest.java
 ##########
 @@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessUtils;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+
+/**
+ * Tests for {@link WorkerResourceSpec}.
+ */
+public class WorkerResourceSpecTest {
 
 Review comment:
   ```suggestion
   public class WorkerResourceSpecTest extends TestLogger {
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 09de8cdeb4d5f82649175e36fb95024f6cf55614 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159244566) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193) 
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 6d44eb6678840e0bd288748962b33ee8539b3d5a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-614595356
 
 
   Thanks @xintongsong for fixing the test. I had one last comment concerning this hotfix.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388879773
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
 ##########
 @@ -272,10 +267,9 @@ private void requestKubernetesPod() {
 	 * Request new pod if pending pods cannot satisfy pending slot requests.
 	 */
 	private void requestKubernetesPodIfRequired() {
-		final int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		final int pendingTaskManagerSlots = numPendingPodRequests * numSlotsPerTaskManager;
+		final int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		if (requiredTaskManagers > numPendingPodRequests) {
 
 Review comment:
   Oops I mix up `requestKubernetesPod` with `requestKubernetesPodIfRequired`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389435941
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   Yes, you are right. We should not require the users to set TM resource related configurations on JM side for standalone. This is the right direction. 
   What i am thinking is when there are some future changes of the `SlotManagerImpl`, we always need to be careful and keep in mind that `WorkerResourceSpec` should be `null` for standalone cluster.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 752470d53f899c7d415ad90e388c765563dd8fae Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160540563) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593) 
   * 80a7fd2aeed32cc6641e5830f28637d5761b7670 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160552877) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-610114796
 
 
   @tillrohrmann 
   
   My concern for standalone clusters generating `WorkerResourceSpec` was that, it requires proper memory configurations and may fail if certain options are absent or conflicts exist. For a standalone cluster, user may only have the memory configurations on the TM machines but not on the JM machine. It is not a good experience to fail the JM process due to the calculation of `WorkerResourceSpec` which is not really needed in such scenarios.
   
   I think you're right, a random/arbitrary `WorkerResourceSpec` should help reduce the complexity. It should not cause any problem since the `StandaloneResourceManager` will ignore the requested `WorkerResourceSpec` anyway.
   
   Thanks again for the review and suggestions. I'll try to address them and update the PR asap.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r404473798
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1054,10 +1054,10 @@ protected abstract void internalDeregisterApplication(
 	 * Allocates a resource using the resource profile.
 	 *
 	 * @param resourceProfile The resource description
-	 * @return Collection of {@link ResourceProfile} describing the launched slots
+	 * @return whether the resource can be allocated
 	 */
 	@VisibleForTesting
-	public abstract Collection<ResourceProfile> startNewWorker(ResourceProfile resourceProfile);
+	public abstract boolean startNewWorker(ResourceProfile resourceProfile);
 
 Review comment:
   True, it is possible that RM won't be able to start a container for given `WorkerResourceSpec`.
   
   I think if the requested resource, to be specific the `TaskExecutorProcessSpec` derived from `WorkerResourceSpec`, is smaller than the Yarn min-allocation, we can simply throw a warning for that. Same for when requested resource is not an integer multiple of min-allocation. The critical part is if the requested resource is larger than the max-allocation, and we should check return false in such cases.
   
   I think these checks should be performed by the RM implementations. I'll add the checks for Yarn in #11353. For K8s, I think there are no such min/max-allocation limits, but I'll double-check on that.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403037763
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -807,23 +811,32 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 		return false;
 	}
 
-	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile requestedSlotResourceProfile) {
+		if (defaultWorkerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
+			return Optional.empty();
+		}
 
 Review comment:
   I think here we are leaking details from the enclosing component into the `SlotManager`. I think the `SlotManager` should not decide for the `ResourceManager` whether it can allocate new resources or not.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f7068628ed9f11c4a4d0675230df530ceed251c5 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155768940) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733) 
   * df11b25dc9094509f24cec5484a6c5b01622d3f8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389238535
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   suggestion: Document `null` semantic of `WorkerResourceSpec`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r401731886
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesResourceManagerFactory.java
 ##########
 @@ -80,4 +84,10 @@ public static KubernetesResourceManagerFactory getInstance() {
 			fatalErrorHandler,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(Configuration configuration) {
+		double fallback = configuration.getDouble(KubernetesConfigOptions.TASK_MANAGER_CPU);
 
 Review comment:
   Out of scope for this PR but at some point we should unify the different CPU options for different deployments.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-604889685
 
 
   Rebased to resolve conflicts.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-614599445
 
 
   @tillrohrmann 
   True, I cherry-picked this hotfix commit from another branch, where `MathUtils.divideRoundU` was not introduced.
   I've updated the PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408959570
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
 ##########
 @@ -267,10 +267,9 @@ private void requestKubernetesPod() {
 	 * Request new pod if pending pods cannot satisfy pending slot requests.
 	 */
 	private void requestKubernetesPodIfRequired() {
-		final int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		final int pendingTaskManagerSlots = numPendingPodRequests * numSlotsPerTaskManager;
+		int requiredTaskManagers = getNumberRequiredTaskManagers();
 
 Review comment:
   ```suggestion
   		final int requiredTaskManagers = getNumberRequiredTaskManagers();
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/152302512) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052) 
   * f7068628ed9f11c4a4d0675230df530ceed251c5 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155768940) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403806981
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/ResourceActions.java
 ##########
 @@ -42,9 +40,9 @@
 	 * Requests to allocate a resource with the given {@link ResourceProfile}.
 	 *
 	 * @param resourceProfile for the to be allocated resource
-	 * @return Collection of {@link ResourceProfile} describing the allocated slots
+	 * @return whether the resource can be allocated
 	 */
-	Collection<ResourceProfile> allocateResource(ResourceProfile resourceProfile);
+	boolean allocateResource(ResourceProfile resourceProfile);
 
 Review comment:
   I'm not sure whether I have correctly understood your question.
   Telling `allocateResource` how big the worker is in the scope of making slot allocation pluggable. In this PR, we have the SM to tell `allocateResource` the `WorkerResourceSpec`, in a subsequence commit rather than this one. This PR does not make the RM implementations to respect the `WorkerResourceSpec` they are told. We make those changes in subsequence PRs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/151939995) 
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/152302512) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 09de8cdeb4d5f82649175e36fb95024f6cf55614 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159244566) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388815917
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   I am also thinking whether we could do better about the way to pass the `DefaultWorkerResourceSpec` to `SlotManagerImpl`. 
   
   Currently, we create in the `XXResourceManager` and pass it to the `ResourceManagerRuntimeServicesConfiguration` -> `SlotManagerConfiguration`. If we could update the flink configuration earlier(especially for the `taskmanager.cpu.cores`) before passing it to the `SlotManagerConfiguration`, then we could move `DefaultWorkerResourceSpec` calculation logics into `SlotManagerConfiguration`. Then the `ResourceManager` is not aware of `DefaultWorkerResourceSpec` since it does not need to know. Also we could avoid the `Nullable` annotation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-614599445
 
 
   @tillrohrmann 
   True, I cherry-picked this hotfix commit from another branch, where `MathUtils.divideRoundUp` was not introduced.
   I've updated the PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * df11b25dc9094509f24cec5484a6c5b01622d3f8 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155803652) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r408964659
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -199,6 +200,14 @@ public int getNumberFreeSlotsOf(InstanceID instanceId) {
 	}
 
 	@Override
+	public Map<WorkerResourceSpec, Integer> getPendingWorkerNums() {
+		final int pendingWorkerNum = (int) Math.ceil((double) pendingSlots.size() / numSlotsPerWorker);
 
 Review comment:
   On the other hand, it complicates the expression a bit. Maybe introducing a name for this function `multipleOfAndLargerOrEqualThan(numSlotsPerWorker, pendingSlots.size())`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * ccdc15f99fc12ed13c1949a36af33ef2d5d806e3 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/159283988) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219) 
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 9528d6ec0d614bf1d66c1908f7679086f0fca961 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389454700
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   @wangyang0918
   True. That's why we annotate it as `@Nullable` and, as suggested by Tison and updated to the PR, clearly documented that a `null` value means in standalone mode.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r404082884
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
 ##########
 @@ -1054,10 +1054,10 @@ protected abstract void internalDeregisterApplication(
 	 * Allocates a resource using the resource profile.
 	 *
 	 * @param resourceProfile The resource description
-	 * @return Collection of {@link ResourceProfile} describing the launched slots
+	 * @return whether the resource can be allocated
 	 */
 	@VisibleForTesting
-	public abstract Collection<ResourceProfile> startNewWorker(ResourceProfile resourceProfile);
+	public abstract boolean startNewWorker(ResourceProfile resourceProfile);
 
 Review comment:
   Ok if I understand it correctly, then this PR relies on the fact that the `RM` and the `SlotManager` both get the same (implicitly or explicitly) `WorkerResourceSpec` assigned, right?
   
   I think it can still happen that the RM won't be able to start exactly a container of the size `WorkerResourceSpec`, even after later changes. Imagine that we are on Yarn were the requested `WorkerResourceSpec` is below the minimum. How should Flink behave in this situation? Should we fail? Should we leave some of the resources unused? Or should we tell the `SlotManager` about it so that it can take the additional resources into account?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-614632389
 
 
   All right, thanks for updating this PR. I'll take it from here and merge it once Travis gives green light.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388860674
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManager.java
 ##########
 @@ -272,10 +267,9 @@ private void requestKubernetesPod() {
 	 * Request new pod if pending pods cannot satisfy pending slot requests.
 	 */
 	private void requestKubernetesPodIfRequired() {
-		final int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		final int pendingTaskManagerSlots = numPendingPodRequests * numSlotsPerTaskManager;
+		final int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		if (requiredTaskManagers > numPendingPodRequests) {
 
 Review comment:
   I can see the return value of `startWorker` is `@return whether the resource can be allocated`
   
   If so, we should also return boolean in this method where false if we don't request a new pod.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388878181
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   @TisonKun
   Please see my reply to Yang for why calling `allocateResource` on standalone setup. For the same reason, we cannot merge this into the `if (!resourceActions.allocateResource(workerResourceSpec))` branch.
   
   Regarding `slotResourceProfile`, `resourceProfile` and `workerResourceSpec`.
   - `slotResourceProfile` describes the resource capacity of a slot.
   - `resourceProfile` describes the resource requirement of a slot request.
   - `workerResourceSpec` describes the resource requirement of a worker (TM) to request.
   I think you're right, these things are indeed hard to understand. How do you think about renaming `resourceProfile` to `requestedSlotResourceProfile`, and `slotResourceProfile` to `defaultSlotResourceProfile`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-602429506
 
 
   @flinkbot run azure

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388872808
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
+			LOG.info(
+				"The amount of cpu cores must be a positive integer on Yarn. Rounding {} up to the closest positive integer {}.",
+				cpuCoresDouble,
+				cpuCoresLong);
+		}
+		if (cpuCoresLong > Integer.MAX_VALUE) {
 
 Review comment:
   @TisonKun Not sure what do you mean. Are you suggesting simplify this code with a `Preconditions.checkstate()`?
   I remember the reason we implemented it like this in `YarnResourceManager` at the first place, is to get an `IllegalConfigurationException` instead of a `IllegalStateException`, for better readability to the users. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 6d44eb6678840e0bd288748962b33ee8539b3d5a Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/159265966) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210) 
   * ccdc15f99fc12ed13c1949a36af33ef2d5d806e3 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159283988) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 9528d6ec0d614bf1d66c1908f7679086f0fca961 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160485130) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550) 
   * 193ccd04ff0c811157bf073f778b5c30b3d80e3d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * df11b25dc9094509f24cec5484a6c5b01622d3f8 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155803652) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744) 
   * 09de8cdeb4d5f82649175e36fb95024f6cf55614 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot commented on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot commented on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595753573
 
 
   Thanks for the review, @wangyang0918 and @TisonKun.
   
   @wangyang0918, regarding where to create `defaultWorkerResourceSpec`:
   - First, it is created in `ActiveResourceManagerFactory`, before creating `SlotManager` and using it to create `ResourceManager`. To that end, I think `ResourceManager` is still not aware of `defaultWorkerResourceSpec`.
   - One reason for not creating `defaultWorkerResourceSpec` in `SlotManagerConfiguration`, is that we need to decide whether to create it or not according to the deployment. We should only create it for active RM setups. See my reply on the detailed comment.
   - Another reason is that, we need to use different configurations per RM to decide the cpu cores of the `defaultWorkerResourceSpec`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389241300
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   @xintongsong thanks for your explanation! That sounds good.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388815917
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   I am also thinking whether we could do better about the way to pass the `DefaultWorkerResourceSpec` to `SlotManagerImpl`. 
   
   Currently, we create in the `XXResourceManager` and pass it to the `ResourceManagerRuntimeServicesConfiguration` -> `SlotManagerConfiguration`. If we could update the flink configuration earlier(especially for the `taskmanager.cpu.cores`) before passing it to the `SlotManagerConfiguration`, then we could move `DefaultWorkerResourceSpec` calculation logics into `SlotManagerConfiguration`. Then the `ResourceManager` is not aware of `DefaultWorkerResourceSpec` since it does not need to know. Also we could avoid the `Nullable` annotation.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389332767
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
 
 Review comment:
   Yes that makes sense. Nullable return value and field has their meaning respectively. It seems we have too many Nullable WorkerResourceSpec parameter(just for passing from here to there), maybe they will be later gathered into one single source?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403789023
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
 
 Review comment:
   It calls to the abstract method `getDefaultCpus`, which is currently overridden differently by different `ActiveResourceManagerFactory` implementations.
   
   Alternatively, we could make this a static method in either `ActiveResourceManagerFactory` of `TaskExecutorProcessUtils`, and make cpu cores an argument of this method. Just not sure how necessary it is.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403032569
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -133,14 +133,17 @@
 	@Nullable
 	private final WorkerResourceSpec defaultWorkerResourceSpec;
 
+	private final int numSlotsPerWorker;
+
 	public SlotManagerImpl(
 			SlotMatchingStrategy slotMatchingStrategy,
 			ScheduledExecutor scheduledExecutor,
 			Time taskManagerRequestTimeout,
 			Time slotRequestTimeout,
 			Time taskManagerTimeout,
 			boolean waitResultConsumedBeforeRelease,
-			@Nullable WorkerResourceSpec defaultWorkerResourceSpec) {
+			@Nullable WorkerResourceSpec defaultWorkerResourceSpec,
+			int numSlotsPerWorker) {
 
 Review comment:
   Maybe it would make sense to pass in the `SlotManagerConfiguration`. If this is possible, then one would not have to add another field to the constructor every time we add a new configuration parameter for the `SlotManager`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403030485
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
 ##########
 @@ -81,4 +83,14 @@ private static Configuration createActiveResourceManagerConfiguration(Configurat
 		ClusterInformation clusterInformation,
 		@Nullable String webInterfaceUrl,
 		ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception;
+
+	protected WorkerResourceSpec createDefaultWorkerResourceSpec(Configuration configuration) {
+		final TaskExecutorProcessSpec taskExecutorProcessSpec = TaskExecutorProcessUtils
+			.newProcessSpecBuilder(configuration)
+			.withCpuCores(getDefaultCpus(configuration))
+			.build();
+		return WorkerResourceSpec.fromTaskExecutorProcessSpec(taskExecutorProcessSpec);
+	}
+
+	protected abstract CPUResource getDefaultCpus(Configuration configuration);
 
 Review comment:
   The same applies to this method. I'm not entirely sure whether this class is the right place for this method. The `ActiveResourceManagerFactory` should not need to know how to derive these things ideally.
   
   Not saying that we have to do it right away but I think the cleaner approach would be to pass a `WorkerResourceSpecFactory` to the `ActiveResourceManagerFactory` or even better, pass simply the default `WorkerResourceSpec` to the constructor of the `ActiveResourceManagerFactory`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388721585
 
 

 ##########
 File path: flink-kubernetes/src/main/java/org/apache/flink/kubernetes/entrypoint/KubernetesResourceManagerFactory.java
 ##########
 @@ -80,4 +84,10 @@ public static KubernetesResourceManagerFactory getInstance() {
 			fatalErrorHandler,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(Configuration configuration) {
 
 Review comment:
   Same here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann closed pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann closed pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

wangyang0918 commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388845338
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
 ##########
 @@ -804,22 +808,31 @@ private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile)
 	}
 
 	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) {
-		final Collection<ResourceProfile> requestedSlots = resourceActions.allocateResource(resourceProfile);
+		if (workerResourceSpec == null) {
+			// standalone mode, cannot allocate resource
 
 Review comment:
   I am also curious about why do we make standalone cluster so different? Before this PR, the `StandaloneResourceManager#startNewWorker` is also called and just return an empty collection.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 09de8cdeb4d5f82649175e36fb95024f6cf55614 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159244566) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193) 
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 6d44eb6678840e0bd288748962b33ee8539b3d5a Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159265966) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r389238604
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/WorkerResourceSpec.java
 ##########
 @@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.resourcemanager;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.resources.CPUResource;
+import org.apache.flink.configuration.MemorySize;
+import org.apache.flink.runtime.clusterframework.TaskExecutorProcessSpec;
+import org.apache.flink.util.Preconditions;
+
+import java.util.Objects;
+
+/**
+ * Resource specification of a worker, mainly used by SlotManager requesting from ResourceManager.
+ */
+public class WorkerResourceSpec {
+	private final CPUResource cpuCores;
+
+	private final MemorySize taskHeapSize;
+
+	private final MemorySize taskOffHeapSize;
+
+	private final MemorySize networkMemSize;
+
+	private final MemorySize managedMemSize;
+
+	public WorkerResourceSpec(
+		CPUResource cpuCores,
+		MemorySize taskHeapSize,
+		MemorySize taskOffHeapSize,
+		MemorySize networkMemSize,
+		MemorySize managedMemSize) {
+
+		this.cpuCores = Preconditions.checkNotNull(cpuCores);
+		this.taskHeapSize = Preconditions.checkNotNull(taskHeapSize);
+		this.taskOffHeapSize = Preconditions.checkNotNull(taskOffHeapSize);
+		this.networkMemSize = Preconditions.checkNotNull(networkMemSize);
+		this.managedMemSize = Preconditions.checkNotNull(managedMemSize);
+	}
+
+	@VisibleForTesting
+	public WorkerResourceSpec(
 
 Review comment:
   Share we have a inner Builder to reduce multiple testing purpose constructor override?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388863553
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
 ##########
 @@ -527,10 +524,9 @@ private FinalApplicationStatus getYarnStatus(ApplicationStatus status) {
 	 * Request new container if pending containers cannot satisfy pending slot requests.
 	 */
 	private void requestYarnContainerIfRequired() {
-		int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		int pendingTaskManagerSlots = numPendingContainerRequests * numSlotsPerTaskManager;
+		int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		if (requiredTaskManagers > numPendingContainerRequests) {
 
 Review comment:
   ditto here as in `KubernetesResourceManager `

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

KarmaGYZ commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388690566
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
 
 Review comment:
   This duplicates with `YarnResourceManager#getCpuCores`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * ccdc15f99fc12ed13c1949a36af33ef2d5d806e3 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/159283988) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219) 
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 9528d6ec0d614bf1d66c1908f7679086f0fca961 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160485130) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot commented on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot commented on issue #11320: [FLINK-16437] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595170974
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0 (Thu Mar 05 11:05:27 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
    * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-16437).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

TisonKun commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388877935
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnResourceManagerFactory.java
 ##########
 @@ -81,4 +91,27 @@ public static YarnResourceManagerFactory getInstance() {
 			webInterfaceUrl,
 			resourceManagerMetricGroup);
 	}
+
+	@Override
+	protected CPUResource getDefaultCpus(final Configuration configuration) {
+		int fallback = configuration.getInteger(YarnConfigOptions.VCORES);
+		double cpuCoresDouble = TaskExecutorProcessUtils.getCpuCoresWithFallback(configuration, fallback).getValue().doubleValue();
+		@SuppressWarnings("NumericCastThatLosesPrecision")
+		long cpuCoresLong = Math.max((long) Math.ceil(cpuCoresDouble), 1L);
+		//noinspection FloatingPointEquality
+		if (cpuCoresLong != cpuCoresDouble) {
+			LOG.info(
+				"The amount of cpu cores must be a positive integer on Yarn. Rounding {} up to the closest positive integer {}.",
+				cpuCoresDouble,
+				cpuCoresLong);
+		}
+		if (cpuCoresLong > Integer.MAX_VALUE) {
 
 Review comment:
   @xintongsong that makes sense.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160540563",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7593",
       "triggerID" : "752470d53f899c7d415ad90e388c765563dd8fae",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7600",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160552877",
       "triggerID" : "80a7fd2aeed32cc6641e5830f28637d5761b7670",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160592479",
       "triggerID" : "7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 7b2346b7cdc46a7c3b490c6c9a8ce6e1727fe10b Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160592479) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7610) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r388870649
 
 

 ##########
 File path: flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java
 ##########
 @@ -527,10 +524,9 @@ private FinalApplicationStatus getYarnStatus(ApplicationStatus status) {
 	 * Request new container if pending containers cannot satisfy pending slot requests.
 	 */
 	private void requestYarnContainerIfRequired() {
-		int requiredTaskManagerSlots = getNumberRequiredTaskManagerSlots();
-		int pendingTaskManagerSlots = numPendingContainerRequests * numSlotsPerTaskManager;
+		int requiredTaskManagers = getNumberRequiredTaskManagers();
 
-		if (requiredTaskManagerSlots > pendingTaskManagerSlots) {
+		if (requiredTaskManagers > numPendingContainerRequests) {
 
 Review comment:
   Same here.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

xintongsong commented on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-609556553

Thanks for the review, @tillrohrmann.

Actually, I have also considered having a special `WorkerResourceSpec.UNKNOWN`. However, I think this is different from `ResourceProfile.UNKNOWN` or `ResourceSpec.UNKNOWN`. For `ResourceProfile`/`ResourceSpec`, `UNKNOWN` means we need some resource but we don't known how many exactly. For `defaultWorkerResourceSpec`, the semantic we actually need for standalone clusters is "the default worker does not exist", or in other words "by default we do not request workers".

My concern is that, `WorkerResourceSpec` is also used as the argument type of `ResourceActions#allocateResource`. With the current design, SM should always request workers from RM with specific worker resource spec. If we add WorkerResourceSpec.UNKNOWN, we have to carefully make sure it is never passed to `allocateResource`.

Combining your concerns and mine, I would propose the following approach.
- Introduce an `DefaultWorkerSlotResourceProvider`, with methods `Optional<WorkerResourceSpec> getDefaultWorkerResourceSpec` and `Optional<ResourceProfile> getDefaultSlotResourceProfile`.
- For standalone clusters, we can pass a special `DefaultWorkerSlotResourceProvider.NO_DEFAULTS` to `SlotManagerImpl`, which returns `Optional.empty()` for both methods. The absent optional value clearly reflects the semantics of "not existing".
- `SlotManagerImpl` no longer needs to assume which RM implementation it works with. It simply not allocate resource from RM if `DefaultWorkerSlotResourceProvider#getDefaultWorkerResourceSpec` returns empty.

WDYT?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [flink] tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

tillrohrmann commented on a change in pull request #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403048662
 
 

 ##########
 File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerFailUnfulfillableTest.java
 ##########
 @@ -52,6 +53,9 @@
  */
 public class SlotManagerFailUnfulfillableTest extends TestLogger {
 
+	private static final WorkerResourceSpec WORKER_RESOURCE_SPEC = new WorkerResourceSpec(
+		100.0, 10000, 10000, 10000, 10000);
 
 Review comment:
   Looking at this line, I would recommend introducing a builder for the `WorkerResourceSpec` and to make the default constructor private.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/152302512) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052) 
   * f7068628ed9f11c4a4d0675230df530ceed251c5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "SUCCESS",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 9528d6ec0d614bf1d66c1908f7679086f0fca961 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160485130) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550) 
   * 193ccd04ff0c811157bf073f778b5c30b3d80e3d Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160507168) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571) 
   * 0e50d9fbea8542a38840bac1150f527f343e5887 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c046c02c0afdfe2c17bc226aa32a249c438978d0 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/152302512) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [flink] flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.

Posted by GitBox <gi...@apache.org>.

flinkbot edited a comment on issue #11320: [FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#issuecomment-595174225
 
 
   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/151939995",
       "triggerID" : "b46d3159b0b1fe2a7f5f3210185c50b5e7252cc0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/flink-ci/flink/builds/152302512",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c046c02c0afdfe2c17bc226aa32a249c438978d0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6052",
       "triggerID" : "596297817",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155768940",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6733",
       "triggerID" : "f7068628ed9f11c4a4d0675230df530ceed251c5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/155803652",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6744",
       "triggerID" : "df11b25dc9094509f24cec5484a6c5b01622d3f8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159244566",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7193",
       "triggerID" : "09de8cdeb4d5f82649175e36fb95024f6cf55614",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8e8aca52ebd0560d829ef3945884817668976804",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8e8aca52ebd0560d829ef3945884817668976804",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7210",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159265966",
       "triggerID" : "6d44eb6678840e0bd288748962b33ee8539b3d5a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/159283988",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7219",
       "triggerID" : "ccdc15f99fc12ed13c1949a36af33ef2d5d806e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "40f6f68a12a74aacf1100dfb81b5bed081cd9a4a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7550",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "status" : "DELETED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160485130",
       "triggerID" : "9528d6ec0d614bf1d66c1908f7679086f0fca961",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "status" : "CANCELED",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160507168",
       "triggerID" : "193ccd04ff0c811157bf073f778b5c30b3d80e3d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "PENDING",
       "url" : "https://travis-ci.com/github/flink-ci/flink/builds/160526443",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584",
       "triggerID" : "0e50d9fbea8542a38840bac1150f527f343e5887",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8e8aca52ebd0560d829ef3945884817668976804 UNKNOWN
   * 40f6f68a12a74aacf1100dfb81b5bed081cd9a4a UNKNOWN
   * 193ccd04ff0c811157bf073f778b5c30b3d80e3d Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160507168) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7571) 
   * 0e50d9fbea8542a38840bac1150f527f343e5887 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160526443) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7584) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services