You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/04/02 08:45:17 UTC
[GitHub] [flink] KarmaGYZ opened a new pull request #11615: [FLINK-16605]
Add max limitation to the total number of slots
KarmaGYZ opened a new pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615
## What is the purpose of the change
Add max limitation to the total number of slots.
## Brief change log
- Introduce "cluster.number-of-slots.max" configuration option with default value MAX_INT
- Make the SlotManager respect the max number of slots, when exceeded, it would not allocate resource anymore.
## Verifying this change
- SlotManagerImplTest#testRespectMaxLimitBeforeAllocateResource
- SlotManagerImplTest#testReleaseResourceWhenExceedMaxLimit
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0a168d7a43629f9b4aa5741b2a43bad4ae8c2237 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159726261) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309)
* 28759e462a28b96201e1ca7f2001676bafa6d473 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0a168d7a43629f9b4aa5741b2a43bad4ae8c2237 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159726261) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407369321
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##########
@@ -76,6 +77,14 @@
.withDeprecatedKeys("yarn.heap-cutoff-min")
.withDescription("Minimum amount of heap memory to remove in Job Master containers, as a safety margin.");
+ @Documentation.Section(Documentation.Sections.EXPERT_SCHEDULING)
+ public static final ConfigOption<Integer> MAX_SLOT_NUM = ConfigOptions
+ .key("slotmanager.number-of-slots.max")
+ .intType()
+ .defaultValue(Integer.MAX_VALUE)
+ .withDescription("Defines the max limitation of the total number of slots. This configuration is meant for batch " +
+ "scenarios. For streaming job, it might cause problems. For standalone cluster, this configuration will not take effect.");
Review comment:
I would suggest the following:
```
Defines the maximum number of slots that the Flink cluster allocates.
This configuration option is meant for limiting the memory consumption for batch workloads.
It is not recommended to configure this option for streaming workloads, which may fail if there are not enough slots.
Note that this configuration option does not take effect for standalone clusters, where how many slots are allocated is not controlled by Flink.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r402719726
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -131,13 +134,15 @@ public SlotManagerImpl(
Time taskManagerRequestTimeout,
Time slotRequestTimeout,
Time taskManagerTimeout,
+ int maxSlotNum,
boolean waitResultConsumedBeforeRelease) {
this.slotMatchingStrategy = Preconditions.checkNotNull(slotMatchingStrategy);
this.scheduledExecutor = Preconditions.checkNotNull(scheduledExecutor);
this.taskManagerRequestTimeout = Preconditions.checkNotNull(taskManagerRequestTimeout);
this.slotRequestTimeout = Preconditions.checkNotNull(slotRequestTimeout);
this.taskManagerTimeout = Preconditions.checkNotNull(taskManagerTimeout);
+ this.maxSlotNum = maxSlotNum;
Review comment:
We should have a sanity check here to make sure `maxSlotNum > 0`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KarmaGYZ commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r406001414
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -375,6 +375,12 @@ public void registerTaskManager(final TaskExecutorConnection taskExecutorConnect
if (taskManagerRegistrations.containsKey(taskExecutorConnection.getInstanceID())) {
reportSlotStatus(taskExecutorConnection.getInstanceID(), initialSlotReport);
} else {
+ if (getNumberRegisteredSlots() + Math.max(getNumberPendingTaskManagerSlots(), numSlotsPerWorker) > maxSlotNum) {
+ LOG.warn("The total number of slots exceeds the max limitation, release the excess resource.");
+ resourceActions.releaseResource(taskExecutorConnection.getInstanceID(), new FlinkException("The total number of slots exceeds the max limitation."));
+ return;
+ }
Review comment:
I think we could edit the contract of `ResourceActions#releaseResource`, let it return whether the release action succeeds. For `StandaloneResourceManager`, it would return false. In this case, it probably makes sense to continue to register the `reportedSlots`. So that, the slot manager need not be aware of whether it's running in a Standalone- or YarnResourceManager. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97223a194aa83228049de648853725b9728b7ccc Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157719880) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975)
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 411969e6faeeaf01a068b67d423a6854e908f328 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160046474) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400)
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160663186",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "778239dfa00a84ab8457d49eedc759b601028676",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632",
"triggerID" : "778239dfa00a84ab8457d49eedc759b601028676",
"triggerType" : "PUSH"
}, {
"hash" : "778239dfa00a84ab8457d49eedc759b601028676",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160682224",
"triggerID" : "778239dfa00a84ab8457d49eedc759b601028676",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160663186) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618)
* 778239dfa00a84ab8457d49eedc759b601028676 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160682224) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 411969e6faeeaf01a068b67d423a6854e908f328 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160046474) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400)
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r402718103
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/ClusterOptions.java
##########
@@ -69,4 +69,11 @@
.text("Enable the slot spread out allocation strategy. This strategy tries to spread out " +
"the slots evenly across all available %s.", code("TaskExecutors"))
.build());
+
+ @Documentation.Section(Documentation.Sections.EXPERT_SCHEDULING)
+ public static final ConfigOption<Integer> MAX_SLOT_NUM = ConfigOptions
+ .key("cluster.number-of-slots.max")
+ .intType()
+ .defaultValue(Integer.MAX_VALUE)
+ .withDescription("Defines the max limitation of the total number of slots.");
Review comment:
I think we should add this to `ResourceManagerOptions` rather than `ClusterOptions`, with key prefix `slotmanager.*`. The option is only used in `SlotManager`, and once FLINK-14106 is finished this might even become a plugin-specific config option.
And I would also mention in the description that this is meant for batch scenarios and might cause problems on streaming jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407373680
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/StandaloneResourceManagerFactory.java
##########
@@ -48,14 +49,18 @@
ClusterInformation clusterInformation,
@Nullable String webInterfaceUrl,
ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception {
+ // The max slot limit should not take effect for standalone cluster, we override the configure in case user
+ // sets this value by mistake.
+ Configuration copiedConfig = new Configuration(configuration);
+ copiedConfig.setInteger(ResourceManagerOptions.MAX_SLOT_NUM, Integer.MAX_VALUE);
Review comment:
nit: I would suggest to have a separated util method `getConfigurationForStandaloneResourceManager` for the configuration overwriting, to encapsulate the details from `StandaloneResourceManagerFactory`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97223a194aa83228049de648853725b9728b7ccc Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157719880) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407409298
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerConfiguration.java
##########
@@ -63,8 +63,10 @@ public SlotManagerConfiguration(
this.slotRequestTimeout = Preconditions.checkNotNull(slotRequestTimeout);
this.taskManagerTimeout = Preconditions.checkNotNull(taskManagerTimeout);
this.waitResultConsumedBeforeRelease = waitResultConsumedBeforeRelease;
- this.slotMatchingStrategy = slotMatchingStrategy;
- this.defaultWorkerResourceSpec = defaultWorkerResourceSpec;
+ this.slotMatchingStrategy = Preconditions.checkNotNull(slotMatchingStrategy);
+ this.defaultWorkerResourceSpec = Preconditions.checkNotNull(defaultWorkerResourceSpec);
+ Preconditions.checkState(numSlotsPerWorker > 0);
+ Preconditions.checkState(maxSlotNum > 0);
Review comment:
It would be better to move the hotfix commit before the main commits if possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KarmaGYZ commented on issue #11615: [FLINK-16605] Add max
limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-611844110
@xintongsong Thanks for the review. I've updated the PR, rebased it on #11320 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160663186",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "778239dfa00a84ab8457d49eedc759b601028676",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632",
"triggerID" : "778239dfa00a84ab8457d49eedc759b601028676",
"triggerType" : "PUSH"
}, {
"hash" : "778239dfa00a84ab8457d49eedc759b601028676",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160682224",
"triggerID" : "778239dfa00a84ab8457d49eedc759b601028676",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* 778239dfa00a84ab8457d49eedc759b601028676 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160682224) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407404631
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -807,16 +813,29 @@ private void fulfillPendingSlotRequestWithPendingTaskManagerSlot(PendingSlotRequ
return Optional.empty();
}
- private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile) {
+ private boolean isFulfillableByRegisteredOrPendingSlots(ResourceProfile resourceProfile) {
for (TaskManagerSlot slot : slots.values()) {
if (slot.getResourceProfile().isMatching(resourceProfile)) {
return true;
}
}
+
+ for (PendingTaskManagerSlot slot : pendingSlots.values()) {
+ if (slot.getResourceProfile().isMatching(resourceProfile)) {
+ return true;
+ }
+ }
+
return false;
}
private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile requestedSlotResourceProfile) {
+ if (getNumberPendingTaskManagerSlots() + getNumberRegisteredSlots() + numSlotsPerWorker > maxSlotNum) {
+ LOG.warn("Could not allocate more slots since the number of slots {} already reach maximum {}.",
Review comment:
We should also mention how many slots trying to allocate.
E.g., say `numSlotsPerWorker` is 3, you might see a log as following.
> Could not allocate more slots since the number of slots 9 already reach maximum 10.
This could be confusing because 9 is not reaching 10.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97223a194aa83228049de648853725b9728b7ccc Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/157719880) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160663186",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 411969e6faeeaf01a068b67d423a6854e908f328 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160046474) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400)
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160663186) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 28759e462a28b96201e1ca7f2001676bafa6d473 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/159755051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97223a194aa83228049de648853725b9728b7ccc Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157719880) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975)
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0a168d7a43629f9b4aa5741b2a43bad4ae8c2237 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97223a194aa83228049de648853725b9728b7ccc Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157719880) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r402719311
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerBuilder.java
##########
@@ -37,6 +38,7 @@ private SlotManagerBuilder() {
this.taskManagerRequestTimeout = TestingUtils.infiniteTime();
this.slotRequestTimeout = TestingUtils.infiniteTime();
this.taskManagerTimeout = TestingUtils.infiniteTime();
+ this.maxSlotNum = Integer.MAX_VALUE;
Review comment:
nit: maybe `MAX_SLOT_NUM.defaultValue()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KarmaGYZ commented on issue #11615: [FLINK-16605] Add max
limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-615076187
Rebased to the latest master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 28759e462a28b96201e1ca7f2001676bafa6d473 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/159755051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318)
* 411969e6faeeaf01a068b67d423a6854e908f328 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/160046474) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400)
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407371673
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/StandaloneResourceManagerFactory.java
##########
@@ -48,14 +49,18 @@
ClusterInformation clusterInformation,
@Nullable String webInterfaceUrl,
ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception {
+ // The max slot limit should not take effect for standalone cluster, we override the configure in case user
+ // sets this value by mistake.
+ Configuration copiedConfig = new Configuration(configuration);
+ copiedConfig.setInteger(ResourceManagerOptions.MAX_SLOT_NUM, Integer.MAX_VALUE);
Review comment:
```suggestion
copiedConfig.copiedConfig.removeConfig(ResourceManagerOptions.MAX_SLOT_NUM);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r402740615
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -375,6 +375,12 @@ public void registerTaskManager(final TaskExecutorConnection taskExecutorConnect
if (taskManagerRegistrations.containsKey(taskExecutorConnection.getInstanceID())) {
reportSlotStatus(taskExecutorConnection.getInstanceID(), initialSlotReport);
} else {
+ if (getNumberRegisteredSlots() + Math.max(getNumberPendingTaskManagerSlots(), numSlotsPerWorker) > maxSlotNum) {
+ LOG.warn("The total number of slots exceeds the max limitation, release the excess resource.");
+ resourceActions.releaseResource(taskExecutorConnection.getInstanceID(), new FlinkException("The total number of slots exceeds the max limitation."));
+ return;
+ }
Review comment:
Not sure about checking on task executors' registering.
I think the max slot limit should not take effect for standalone clusters, because the number of task executors launched is not controlled by RM. We should also mention that in the config option description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160663186",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160663186) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97223a194aa83228049de648853725b9728b7ccc Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/157719880) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975)
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0a168d7a43629f9b4aa5741b2a43bad4ae8c2237 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159726261) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0a168d7a43629f9b4aa5741b2a43bad4ae8c2237 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159726261) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309)
* 28759e462a28b96201e1ca7f2001676bafa6d473 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/159755051) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11615: [FLINK-16605] Add max
limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607709911
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 97223a194aa83228049de648853725b9728b7ccc (Thu Apr 02 08:47:28 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407408627
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImplTest.java
##########
@@ -1540,6 +1555,85 @@ public void testSpreadOutSlotAllocationStrategy() throws Exception {
}
}
+ /**
+ * Test that the slot manager respect the max limitation of the number of slots when allocate new resource.
+ */
+ @Test
+ public void testRespectMaxLimitBeforeAllocateResource() throws Exception {
+ final int numberSlots = 1;
+ final int maxSlotNum = 1;
+
+ final ResourceManagerId resourceManagerId = ResourceManagerId.generate();
+ final JobID jobId = new JobID();
+
+ final AtomicInteger resourceRequests = new AtomicInteger(0);
+ ResourceActions resourceManagerActions = new TestingResourceActionsBuilder()
+ .setAllocateResourceFunction(
+ ignored -> {
+ resourceRequests.incrementAndGet();
+ return true;
+ })
+ .build();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(ResourceManagerOptions.MAX_SLOT_NUM, maxSlotNum);
+ configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, numberSlots);
+
+ try (SlotManagerImpl slotManager = createSlotManager(resourceManagerId, resourceManagerActions, configuration)) {
+
+ assertTrue("The slot request should be accepted", slotManager.registerSlotRequest(createSlotRequest(jobId)));
+ assertThat(resourceRequests.get(), is(1));
+
+ // The second slot request should not try to allocate a new resource because of the max limitation.
+ assertTrue("The slot request should be accepted", slotManager.registerSlotRequest(createSlotRequest(jobId)));
+ assertThat(resourceRequests.get(), is(1));
+ }
+ }
+
+ /**
+ * Test that the slot manager release resource when the number of slots exceed max limit when new TaskExecutor registered.
+ */
+ @Test
+ public void testReleaseResourceWhenExceedMaxLimit() throws Exception {
Review comment:
```suggestion
public void testMaxSlotLimitRegisterResource() throws Exception {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KarmaGYZ commented on issue #11615: [FLINK-16605] Add max
limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
KarmaGYZ commented on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-613262503
Thanks for the review, @xintongsong . I've addressed your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "CANCELED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 411969e6faeeaf01a068b67d423a6854e908f328 Travis: [CANCELED](https://travis-ci.com/github/flink-ci/flink/builds/160046474) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400)
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160663186",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160663186) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r406571178
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -375,6 +375,12 @@ public void registerTaskManager(final TaskExecutorConnection taskExecutorConnect
if (taskManagerRegistrations.containsKey(taskExecutorConnection.getInstanceID())) {
reportSlotStatus(taskExecutorConnection.getInstanceID(), initialSlotReport);
} else {
+ if (getNumberRegisteredSlots() + Math.max(getNumberPendingTaskManagerSlots(), numSlotsPerWorker) > maxSlotNum) {
+ LOG.warn("The total number of slots exceeds the max limitation, release the excess resource.");
+ resourceActions.releaseResource(taskExecutorConnection.getInstanceID(), new FlinkException("The total number of slots exceeds the max limitation."));
+ return;
+ }
Review comment:
It seems to me that this approach implicitly assumes that `releaseResource` returns false means running on a standalone cluster, which is not always true. Without this assumption, continuing registering the slot when the release action failed seems quite against intuition.
Instead of let the slot manager behaves differently according to the release action result, I would suggest to simply pass in infinite large values for the max limits when creating the slot manager. In this way, slot manager should not need to behave differently on standalone and active resource managers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r402744613
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -793,16 +799,26 @@ private void fulfillPendingSlotRequestWithPendingTaskManagerSlot(PendingSlotRequ
return Optional.empty();
}
- private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile) {
+ private boolean isFulfillableByRegisteredOrPendingSlots(ResourceProfile resourceProfile) {
for (TaskManagerSlot slot : slots.values()) {
if (slot.getResourceProfile().isMatching(resourceProfile)) {
return true;
}
}
+
+ for (PendingTaskManagerSlot slot : pendingSlots.values()) {
+ if (slot.getResourceProfile().isMatching(resourceProfile)) {
+ return true;
+ }
+ }
+
return false;
}
private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) throws ResourceManagerException {
+ if (getNumberPendingTaskManagerSlots() + getNumberRegisteredSlots() + numSlotsPerWorker > maxSlotNum) {
+ return Optional.empty();
+ }
Review comment:
I think this makes the SM/RM contract a bit unclear.
Before this PR, SM requests individual slots from RM and is not aware of `numSlotsPerWorker`.
With this change, SM is aware of slots per worker only for the max slots limit check, while for other logics (e.g., generating pending slots) it still pretend not aware of slots per worker.
I would suggest to either move this check to the RM side, or base this PR on #11320, where we change the contract that SM request workers from RM and is aware of slots per worker in a consistent way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0a168d7a43629f9b4aa5741b2a43bad4ae8c2237 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/159726261) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407396644
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -389,6 +389,12 @@ public void registerTaskManager(final TaskExecutorConnection taskExecutorConnect
if (taskManagerRegistrations.containsKey(taskExecutorConnection.getInstanceID())) {
reportSlotStatus(taskExecutorConnection.getInstanceID(), initialSlotReport);
} else {
+ if (getNumberRegisteredSlots() + Math.max(getNumberPendingTaskManagerSlots(), numSlotsPerWorker) > maxSlotNum) {
Review comment:
Moreover, we should not assumes that the number of slots on the registered task executor is `numSlotsPerWorker`. We should get the number of registered slots from the slot report.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11615: [FLINK-16605] Add
max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-612838446
Thanks for updating the PR, @KarmaGYZ. I have left some comments. Please take a look.
My main concern is regarding checking the max limit for registered task executors.
BTW, there seems to be some conflicts against the master branch. Could you please also look into that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "FAILURE",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160663186",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "778239dfa00a84ab8457d49eedc759b601028676",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "778239dfa00a84ab8457d49eedc759b601028676",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/160663186) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618)
* 778239dfa00a84ab8457d49eedc759b601028676 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 28759e462a28b96201e1ca7f2001676bafa6d473 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/159755051) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318)
* 411969e6faeeaf01a068b67d423a6854e908f328 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r402745707
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -793,16 +799,26 @@ private void fulfillPendingSlotRequestWithPendingTaskManagerSlot(PendingSlotRequ
return Optional.empty();
}
- private boolean isFulfillableByRegisteredSlots(ResourceProfile resourceProfile) {
+ private boolean isFulfillableByRegisteredOrPendingSlots(ResourceProfile resourceProfile) {
for (TaskManagerSlot slot : slots.values()) {
if (slot.getResourceProfile().isMatching(resourceProfile)) {
return true;
}
}
+
+ for (PendingTaskManagerSlot slot : pendingSlots.values()) {
+ if (slot.getResourceProfile().isMatching(resourceProfile)) {
+ return true;
+ }
+ }
+
return false;
}
private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile resourceProfile) throws ResourceManagerException {
+ if (getNumberPendingTaskManagerSlots() + getNumberRegisteredSlots() + numSlotsPerWorker > maxSlotNum) {
+ return Optional.empty();
+ }
Review comment:
In addition, we should add warning logs explicitly show that slots are not allocated because of reaching the max limit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407407657
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImplTest.java
##########
@@ -1540,6 +1555,85 @@ public void testSpreadOutSlotAllocationStrategy() throws Exception {
}
}
+ /**
+ * Test that the slot manager respect the max limitation of the number of slots when allocate new resource.
+ */
+ @Test
+ public void testRespectMaxLimitBeforeAllocateResource() throws Exception {
+ final int numberSlots = 1;
+ final int maxSlotNum = 1;
+
+ final ResourceManagerId resourceManagerId = ResourceManagerId.generate();
+ final JobID jobId = new JobID();
+
+ final AtomicInteger resourceRequests = new AtomicInteger(0);
+ ResourceActions resourceManagerActions = new TestingResourceActionsBuilder()
+ .setAllocateResourceFunction(
+ ignored -> {
+ resourceRequests.incrementAndGet();
+ return true;
+ })
+ .build();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(ResourceManagerOptions.MAX_SLOT_NUM, maxSlotNum);
+ configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, numberSlots);
+
+ try (SlotManagerImpl slotManager = createSlotManager(resourceManagerId, resourceManagerActions, configuration)) {
+
+ assertTrue("The slot request should be accepted", slotManager.registerSlotRequest(createSlotRequest(jobId)));
+ assertThat(resourceRequests.get(), is(1));
+
+ // The second slot request should not try to allocate a new resource because of the max limitation.
+ assertTrue("The slot request should be accepted", slotManager.registerSlotRequest(createSlotRequest(jobId)));
+ assertThat(resourceRequests.get(), is(1));
+ }
+ }
+
+ /**
+ * Test that the slot manager release resource when the number of slots exceed max limit when new TaskExecutor registered.
+ */
+ @Test
+ public void testReleaseResourceWhenExceedMaxLimit() throws Exception {
+ final int numberSlots = 1;
+ final int maxSlotNum = 1;
+ final ResourceManagerId resourceManagerId = ResourceManagerId.generate();
+
+ final CompletableFuture<InstanceID> releasedResourceFuture = new CompletableFuture<>();
+ ResourceActions resourceManagerActions = new TestingResourceActionsBuilder()
+ .setReleaseResourceConsumer((instanceID, e) -> releasedResourceFuture.complete(instanceID))
+ .build();
+
+ final Configuration configuration = new Configuration();
+ configuration.setInteger(ResourceManagerOptions.MAX_SLOT_NUM, maxSlotNum);
+ configuration.setInteger(TaskManagerOptions.NUM_TASK_SLOTS, numberSlots);
+
+ final TaskExecutorGateway taskExecutorGateway1 = new TestingTaskExecutorGatewayBuilder().createTestingTaskExecutorGateway();
+ final TaskExecutorGateway taskExecutorGateway2 = new TestingTaskExecutorGatewayBuilder().createTestingTaskExecutorGateway();
+ final ResourceID resourceId1 = ResourceID.generate();
+ final ResourceID resourceId2 = ResourceID.generate();
+ final TaskExecutorConnection taskManagerConnection1 = new TaskExecutorConnection(resourceId1, taskExecutorGateway1);
+ final TaskExecutorConnection taskManagerConnection2 = new TaskExecutorConnection(resourceId2, taskExecutorGateway2);
+
+ final SlotID slotId1 = new SlotID(resourceId1, 0);
+ final SlotID slotId2 = new SlotID(resourceId1, 0);
+ final SlotStatus slotStatus1 = new SlotStatus(slotId1, ResourceProfile.UNKNOWN);
+ final SlotStatus slotStatus2 = new SlotStatus(slotId2, ResourceProfile.UNKNOWN);
+ final SlotReport slotReport1 = new SlotReport(Collections.singletonList(slotStatus1));
+ final SlotReport slotReport2 = new SlotReport(Collections.singletonList(slotStatus2));
+
+ try (SlotManagerImpl slotManager = createSlotManager(resourceManagerId, resourceManagerActions, configuration)) {
+ slotManager.registerTaskManager(taskManagerConnection1, slotReport1);
+ slotManager.registerTaskManager(taskManagerConnection2, slotReport2);
+
+ assertTrue("The number registered slots does not equal the expected number.", 1 == slotManager.getNumberRegisteredSlots());
Review comment:
Better to use `assertThat(xxx, is(xxx))`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11615: [FLINK-16605]
Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/157719880",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6975",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
}, {
"hash" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2db316d977d0e790de8fa98b27dd219b68abb136",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7309",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159726261",
"triggerID" : "0a168d7a43629f9b4aa5741b2a43bad4ae8c2237",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/159755051",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7318",
"triggerID" : "28759e462a28b96201e1ca7f2001676bafa6d473",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7400",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "411969e6faeeaf01a068b67d423a6854e908f328",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160046474",
"triggerID" : "411969e6faeeaf01a068b67d423a6854e908f328",
"triggerType" : "PUSH"
}, {
"hash" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b0111ba893c9ecc7632394fc59e455f8d5c9db7",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160663186",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7618",
"triggerID" : "cf7728e0b162ed6bb27c75aa2d02ca9be7e2a6c4",
"triggerType" : "PUSH"
}, {
"hash" : "778239dfa00a84ab8457d49eedc759b601028676",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632",
"triggerID" : "778239dfa00a84ab8457d49eedc759b601028676",
"triggerType" : "PUSH"
}, {
"hash" : "778239dfa00a84ab8457d49eedc759b601028676",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/160682224",
"triggerID" : "778239dfa00a84ab8457d49eedc759b601028676",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2db316d977d0e790de8fa98b27dd219b68abb136 UNKNOWN
* 0b0111ba893c9ecc7632394fc59e455f8d5c9db7 UNKNOWN
* 778239dfa00a84ab8457d49eedc759b601028676 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/160682224) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=7632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407382464
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImpl.java
##########
@@ -389,6 +389,12 @@ public void registerTaskManager(final TaskExecutorConnection taskExecutorConnect
if (taskManagerRegistrations.containsKey(taskExecutorConnection.getInstanceID())) {
reportSlotStatus(taskExecutorConnection.getInstanceID(), initialSlotReport);
} else {
+ if (getNumberRegisteredSlots() + Math.max(getNumberPendingTaskManagerSlots(), numSlotsPerWorker) > maxSlotNum) {
Review comment:
I think this `if` condition assumes that the pending slots can be consumed by the new registered slots. In other words, the resource profiles of the registered slots matches the pending slots exactly. I think this is a quite strong assumption. It might be true at the moment, but will weaken the generality of `SlotManagerImpl`.
I would suggest to check whether the registered slots can be matched to the pending slots here, and decide whether to release the registered task executor based on the checkin gresult.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407371314
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/StandaloneResourceManagerFactory.java
##########
@@ -48,14 +49,18 @@
ClusterInformation clusterInformation,
@Nullable String webInterfaceUrl,
ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception {
+ // The max slot limit should not take effect for standalone cluster, we override the configure in case user
+ // sets this value by mistake.
Review comment:
I would suggest to add a warning log saying that the configured value (if any) is ignored.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r402736479
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImplTest.java
##########
@@ -1293,7 +1293,18 @@ private SlotRequest createSlotRequest(JobID jobId, ResourceProfile resourceProfi
}
private SlotManagerImpl createSlotManager(ResourceManagerId resourceManagerId, ResourceActions resourceManagerActions) {
- SlotManagerImpl slotManager = SlotManagerBuilder.newBuilder().build();
+ return createSlotManager(resourceManagerId, resourceManagerActions, 1);
+ }
+
+ private SlotManagerImpl createSlotManager(ResourceManagerId resourceManagerId, ResourceActions resourceManagerActions, int numSlotsPerWorker) {
+ return createSlotManager(resourceManagerId, resourceManagerActions, numSlotsPerWorker, Integer.MAX_VALUE);
+ }
+
+ private SlotManagerImpl createSlotManager(ResourceManagerId resourceManagerId, ResourceActions resourceManagerActions, int numSlotsPerWorker, int maxSlotNum) {
Review comment:
I think we should try to avoid such deep nested methods if possible.
Since both `numSlotsPerWorker` and `maxSlotNum` can be read from a `Configuration`, I think keeping only two `createSlotManager` methods (one with `Configuration` and one without) should be enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11615: [FLINK-16605] Add max
limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-607717339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "97223a194aa83228049de648853725b9728b7ccc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "97223a194aa83228049de648853725b9728b7ccc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97223a194aa83228049de648853725b9728b7ccc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong removed a comment on issue #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong removed a comment on issue #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#issuecomment-612838446
Thanks for updating the PR, @KarmaGYZ. I have left some comments. Please take a look.
My main concern is regarding checking the max limit for registered task executors.
BTW, there seems to be some conflicts against the master branch. Could you please also look into that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407370953
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/StandaloneResourceManagerFactory.java
##########
@@ -48,14 +49,18 @@
ClusterInformation clusterInformation,
@Nullable String webInterfaceUrl,
ResourceManagerMetricGroup resourceManagerMetricGroup) throws Exception {
+ // The max slot limit should not take effect for standalone cluster, we override the configure in case user
+ // sets this value by mistake.
Review comment:
```suggestion
// The max slot limit should not take effect for standalone cluster, we overwrite the configure in case user
// sets this value by mistake.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on a change in pull request #11615:
[FLINK-16605] Add max limitation to the total number of slots
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #11615: [FLINK-16605] Add max limitation to the total number of slots
URL: https://github.com/apache/flink/pull/11615#discussion_r407408525
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManagerImplTest.java
##########
@@ -1540,6 +1555,85 @@ public void testSpreadOutSlotAllocationStrategy() throws Exception {
}
}
+ /**
+ * Test that the slot manager respect the max limitation of the number of slots when allocate new resource.
+ */
+ @Test
+ public void testRespectMaxLimitBeforeAllocateResource() throws Exception {
Review comment:
```suggestion
public void testMaxSlotLimitAllocateResource() throws Exception {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services