You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/05/19 18:47:57 UTC
[GitHub] [flink] zhuzhurk opened a new pull request #12256: [FLINK-17018][runtime][WIP] DefaultExecutionSlotAllocator allocates slots in bulks
zhuzhurk opened a new pull request #12256:
URL: https://github.com/apache/flink/pull/12256
## What is the purpose of the change
Bulk slot allocation is needed to check whether a bulk of slot requests can be fulfilled at the same time. See [FLIP-119|Bulk Slot Allocation](https://cwiki.apache.org/confluence/display/FLINK/FLIP-119+Pipelined+Region+Scheduling#FLIP-119PipelinedRegionScheduling-BulkSlotAllocation).
The dynamic slot sharing makes it hard to implement bulk slot allocation.
So we will ignore slot sharing in the first version. Co-location constraints are not supported as a result.
Intra-bulk input location preferences will also be ignored, otherwise a bulk allocation can be blocked on waiting for producers in the same bulk to be assigned slots first.
## Brief change log
- *Enables SlotPoolImpl to check whether a bulk of slot requests is fulfillable*
- *Enables SchedulerImpl for bulk slot allocation*
- *Enables SlotProviderStrategy for bulk slot allocation*
- *Change DefaultExecutionSlotAllocator to use bulk slot allocation when it is pipelined region scheduling*
## Verifying this change
This change added tests and can be verified as follows:
- *Added tests in SlotPoolImplTest*
- *Added SlotOccupationTest*
- *Added BulkSlotAllocationTest*
- *[WIP] Added DefaultExecutionSlotAllocatorBulkSlotAllocationTest*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (**yes** / no / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r431939772
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eba9c31264c2a0fe9f5bf955ba21de081bd1aa25 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858)
* 5607dc27dad87a19fe195670b6e216bc2850cb2e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438076660
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
Review comment:
Ok I see. `AbstractExecutionSlotAllocator#getSlotProfileFuture()` is added back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916)
* cce8477d3eae8bad8b4109fa1f85e810220b6d39 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117)
* 7f16b12b9c89b723a6c201da252d65e259a53c84 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2aa39a469f86ceae75eafb75194f7c7f82e3a6cc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422)
* 8ba2a2eb65072f71c66055c3f86549a061eb1514 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432272564
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/SlotProviderStrategy.java
##########
@@ -56,6 +62,24 @@
ScheduledUnit scheduledUnit,
SlotProfile slotProfile);
+ /**
+ * Allocates a bulk of physical slots.
+ *
+ * @param physicalSlotRequests requests for physical slots
+ * @return future of the allocation which will be completed normally only when all the requests are fulfilled
+ */
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
Review comment:
done by make `OneSlotPerExecutionSlotAllocator ` interact with `SloProvider` directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r437985809
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
Review comment:
I meant at least to factor out `getSlotProfileFuture` into a separate private method inside `OneSlotPerExecutionSlotAllocator` and another one inside `DefaultExecutionSlotAllocator `. This would reduce the big method `createPhysicalSlotRequestFutures`.
The next idea was to deduplicate by adding what to use for `physicalSlotResourceProfile` in `SlotProfile.priorAllocation`:
```
AbstractExecutionSlotAllocator#getSlotProfileFuture(..., ResourceProfile physicalSlotResourceProfile)
```
This might be too much.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements);
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = slotExecutionVertexAssignments
+ .stream()
+ .collect(
+ Collectors.toMap(
+ SlotExecutionVertexAssignment::getExecutionVertexId,
+ SlotExecutionVertexAssignment::getSlotRequestId));
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments);
+
+ return Collections.unmodifiableList(slotExecutionVertexAssignments);
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ (slotRequestId, throwable) -> slotProvider.cancelSlotRequest(slotRequestId, throwable));
Review comment:
```suggestion
slotProvider::cancelSlotRequest);
```
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/ExecutionSlotAllocatorTestUtils.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Test utils for {@link ExecutionSlotAllocator}.
+ */
+public class ExecutionSlotAllocatorTestUtils {
Review comment:
```suggestion
class ExecutionSlotAllocatorTestUtils {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438659428
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements);
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = slotExecutionVertexAssignments
+ .stream()
+ .collect(
+ Collectors.toMap(
+ SlotExecutionVertexAssignment::getExecutionVertexId,
+ SlotExecutionVertexAssignment::getSlotRequestId));
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments);
+
+ return Collections.unmodifiableList(slotExecutionVertexAssignments);
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ (slotRequestId, throwable) -> slotProvider.cancelSlotRequest(slotRequestId, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(executionVertexId, allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotWillBeOccupiedIndefinitely);
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments) {
+
+ final Map<SlotRequestId, SlotExecutionVertexAssignment> requestToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getSlotRequestId, Function.identity()));
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProvider.allocatePhysicalSlots(physicalSlotRequests, allocationTimeout))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final SlotExecutionVertexAssignment assignment = requestToAssignmentMapping.get(slotRequestId);
+
+ checkState(assignment != null);
+
+ final LogicalSlot logicalSlot = SingleLogicalSlot.allocateFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot(),
+ Locality.UNKNOWN,
+ slotOwner,
+ slotWillBeOccupiedIndefinitely);
+ assignment.getLogicalSlotFuture().complete(logicalSlot);
+ }
+ })
+ .exceptionally(ex -> {
+ slotExecutionVertexAssignments.forEach(
+ assignment -> assignment.getLogicalSlotFuture().completeExceptionally(ex));
+ return null;
+ });
+ }
+
+ private class OneSlotPerExecutionSlotAllocatorSlotOwner implements SlotOwner {
+
+ @Override
+ public void returnLogicalSlot(LogicalSlot logicalSlot) {
+ slotProvider.cancelSlotRequest(
+ logicalSlot.getSlotRequestId(),
+ new FlinkException("Slot is being returned to OneSlotPerExecutionSlotAllocator."));
+ }
+ }
Review comment:
```suggestion
@Override
public void returnLogicalSlot(LogicalSlot logicalSlot) {
slotProvider.cancelSlotRequest(
logicalSlot.getSlotRequestId(),
new FlinkException("Slot is being returned to OneSlotPerExecutionSlotAllocator."));
}
```
Maybe it is just easier to implement `SlotOwner` in `OneSlotPerExecutionSlotAllocator` and pass `this` to `SingleLogicalSlot.allocateFromPhysicalSlot`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -36,41 +35,28 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
-import java.util.Objects;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
-import java.util.stream.Collectors;
import static org.apache.flink.util.Preconditions.checkNotNull;
-import static org.apache.flink.util.Preconditions.checkState;
/**
* Default {@link ExecutionSlotAllocator} which will use {@link SlotProvider} to allocate slots and
* keep the unfulfilled requests for further cancellation.
*/
-public class DefaultExecutionSlotAllocator implements ExecutionSlotAllocator {
+public class DefaultExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
private static final Logger LOG = LoggerFactory.getLogger(DefaultExecutionSlotAllocator.class);
- /**
- * Store the uncompleted slot assignments.
- */
- private final Map<ExecutionVertexID, SlotExecutionVertexAssignment> pendingSlotAssignments;
-
private final SlotProviderStrategy slotProviderStrategy;
- private final PreferredLocationsRetriever preferredLocationsRetriever;
-
public DefaultExecutionSlotAllocator(
final SlotProviderStrategy slotProviderStrategy,
final PreferredLocationsRetriever preferredLocationsRetriever) {
- this.slotProviderStrategy = checkNotNull(slotProviderStrategy);
- this.preferredLocationsRetriever = checkNotNull(preferredLocationsRetriever);
- pendingSlotAssignments = new HashMap<>();
+ super(preferredLocationsRetriever);
Review comment:
nit: constructor can be package-private
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/AbstractExecutionSlotAllocator.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiConsumer;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Base class for all {@link ExecutionSlotAllocator}. It is responsible to allocate slots for tasks and
+ * keep the unfulfilled slot requests for further cancellation.
+ */
+abstract class AbstractExecutionSlotAllocator implements ExecutionSlotAllocator {
+
+ private final Map<ExecutionVertexID, SlotExecutionVertexAssignment> pendingSlotAssignments;
+
+ private final PreferredLocationsRetriever preferredLocationsRetriever;
+
+ protected AbstractExecutionSlotAllocator(final PreferredLocationsRetriever preferredLocationsRetriever) {
+ this.preferredLocationsRetriever = checkNotNull(preferredLocationsRetriever);
+ this.pendingSlotAssignments = new HashMap<>();
+ }
+
+ @Override
+ public void cancel(final ExecutionVertexID executionVertexId) {
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment = pendingSlotAssignments.get(executionVertexId);
+ if (slotExecutionVertexAssignment != null) {
+ slotExecutionVertexAssignment.getLogicalSlotFuture().cancel(false);
+ }
+ }
+
+ protected void validateSchedulingRequirements(
Review comment:
```suggestion
void validateSchedulingRequirements(
```
nit: not sure, why we need `protected` for methods in `AbstractExecutionSlotAllocator` if it is used only within the package
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocatorTest.java
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraint;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroup;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotTestUtils.createPhysicalSlot;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.createSchedulingRequirements;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.findSlotAssignmentByExecutionVertexId;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for {@link OneSlotPerExecutionSlotAllocator}.
+ */
+public class OneSlotPerExecutionSlotAllocatorTest extends TestLogger {
+
+ private TestingBulkSlotProvider slotProvider;
+
+ @Before
+ public void setUp() throws Exception {
+ slotProvider = new TestingBulkSlotProvider();
+ }
+
+ @Test
+ public void testSucceededSlotAllocation() {
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(1));
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getExecutionVertexId(), equalTo(executionVertexID));
+ assertThat(slotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(false));
+ }
+
+ @Test
+ public void testFailedSlotAllocation() {
+ final OneSlotPerExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ slotProvider.forceFailingSlotAllocation();
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(true));
+ assertThat(executionSlotAllocator.getPendingSlotAssignments().keySet(), hasSize(0));
+ assertThat(slotProvider.getCancelledSlotRequestIds(), contains(slotAssignment.getSlotRequestId()));
+ }
+
+ @Test
+ public void testInterBulkInputLocationPreferencesAreRespected() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForProducer =
+ createSchedulingRequirements(producerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForProducer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForProducer);
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignmentsForProducer);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+
+ inputsLocationsRetriever.markScheduled(consumerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForConsumer =
+ createSchedulingRequirements(consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForConsumer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForConsumer);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignmentsForConsumer);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(false));
+
+ inputsLocationsRetriever.assignTaskManagerLocation(producerId);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testIntraBulkInputLocationPreferencesDoNotBlockAllocation() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ inputsLocationsRetriever.markScheduled(consumerId);
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(producerId, consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(2));
+
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignments);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignments);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testCreatedSlotRequests() {
+ final ExecutionVertexID executionVertexId = new ExecutionVertexID(new JobVertexID(), 0);
+ final AllocationID allocationId = new AllocationID();
+ final SlotSharingGroupId sharingGroupId = new SlotSharingGroupId();
+ final ResourceProfile taskResourceProfile = ResourceProfile.fromResources(0.5, 250);
+ final ResourceProfile physicalSlotResourceProfile = ResourceProfile.fromResources(1.0, 300);
+ final TaskManagerLocation taskManagerLocation = new LocalTaskManagerLocation();
+
+ final TestingStateLocationRetriever stateLocationRetriever = new TestingStateLocationRetriever();
+ stateLocationRetriever.setStateLocation(executionVertexId, taskManagerLocation);
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ stateLocationRetriever,
+ new TestingInputsLocationsRetriever.Builder().build());
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Arrays.asList(
Review comment:
```suggestion
final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Collections.singletonList(
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -77,22 +75,20 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
- executionVertexId,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
+ final CompletableFuture<SlotProfile> slotProfileFuture = getSlotProfileFuture(
+ schedulingRequirements,
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ Collections.emptySet(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile -> slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
Review comment:
nit, it would be nice to have:
```
private CompletableFuture<LogicalSlot> allocateSlot(
Set<AllocationID> allPreviousAllocationIds,
ExecutionVertexSchedulingRequirements schedulingRequirements) {
final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
final SlotRequestId slotRequestId = new SlotRequestId();
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
final CompletableFuture<SlotProfile> slotProfileFuture = getSlotProfileFuture(
schedulingRequirements,
schedulingRequirements.getPhysicalSlotResourceProfile(),
Collections.emptySet(),
allPreviousAllocationIds);
return slotProfileFuture.thenCompose(
slotProfile -> slotProviderStrategy.allocateSlot(
slotRequestId,
new ScheduledUnit(
executionVertexId,
schedulingRequirements.getSlotSharingGroupId(),
schedulingRequirements.getCoLocationConstraint()),
slotProfile));
}
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/AbstractExecutionSlotAllocator.java
##########
@@ -94,7 +96,26 @@ protected SlotExecutionVertexAssignment createAndRegisterSlotExecutionVertexAssi
return slotExecutionVertexAssignment;
}
- protected CompletableFuture<Collection<TaskManagerLocation>> calculatePreferredLocations(
+ protected CompletableFuture<SlotProfile> getSlotProfileFuture(
+ final ExecutionVertexSchedulingRequirements schedulingRequirements,
+ final ResourceProfile physicalSlotResourceProfile,
+ final Set<ExecutionVertexID> producersToIgnore,
+ final Set<AllocationID> allPreviousAllocationIds) {
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(schedulingRequirements.getExecutionVertexId(), producersToIgnore);
+
+ return preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ physicalSlotResourceProfile,
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+ }
+
+ private CompletableFuture<Collection<TaskManagerLocation>> calculatePreferredLocations(
Review comment:
If it is a private method now, I guess we do not need it?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocatorTest.java
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraint;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroup;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotTestUtils.createPhysicalSlot;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.createSchedulingRequirements;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.findSlotAssignmentByExecutionVertexId;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for {@link OneSlotPerExecutionSlotAllocator}.
+ */
+public class OneSlotPerExecutionSlotAllocatorTest extends TestLogger {
+
+ private TestingBulkSlotProvider slotProvider;
+
+ @Before
+ public void setUp() throws Exception {
+ slotProvider = new TestingBulkSlotProvider();
+ }
+
+ @Test
+ public void testSucceededSlotAllocation() {
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(1));
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getExecutionVertexId(), equalTo(executionVertexID));
+ assertThat(slotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(false));
+ }
+
+ @Test
+ public void testFailedSlotAllocation() {
+ final OneSlotPerExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ slotProvider.forceFailingSlotAllocation();
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(true));
+ assertThat(executionSlotAllocator.getPendingSlotAssignments().keySet(), hasSize(0));
+ assertThat(slotProvider.getCancelledSlotRequestIds(), contains(slotAssignment.getSlotRequestId()));
+ }
+
+ @Test
+ public void testInterBulkInputLocationPreferencesAreRespected() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForProducer =
+ createSchedulingRequirements(producerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForProducer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForProducer);
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignmentsForProducer);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+
+ inputsLocationsRetriever.markScheduled(consumerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForConsumer =
+ createSchedulingRequirements(consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForConsumer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForConsumer);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignmentsForConsumer);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(false));
+
+ inputsLocationsRetriever.assignTaskManagerLocation(producerId);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testIntraBulkInputLocationPreferencesDoNotBlockAllocation() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ inputsLocationsRetriever.markScheduled(consumerId);
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(producerId, consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(2));
+
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignments);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignments);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testCreatedSlotRequests() {
+ final ExecutionVertexID executionVertexId = new ExecutionVertexID(new JobVertexID(), 0);
+ final AllocationID allocationId = new AllocationID();
+ final SlotSharingGroupId sharingGroupId = new SlotSharingGroupId();
+ final ResourceProfile taskResourceProfile = ResourceProfile.fromResources(0.5, 250);
+ final ResourceProfile physicalSlotResourceProfile = ResourceProfile.fromResources(1.0, 300);
+ final TaskManagerLocation taskManagerLocation = new LocalTaskManagerLocation();
+
+ final TestingStateLocationRetriever stateLocationRetriever = new TestingStateLocationRetriever();
+ stateLocationRetriever.setStateLocation(executionVertexId, taskManagerLocation);
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ stateLocationRetriever,
+ new TestingInputsLocationsRetriever.Builder().build());
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Arrays.asList(
+ new ExecutionVertexSchedulingRequirements.Builder()
+ .withExecutionVertexId(executionVertexId)
+ .withPreviousAllocationId(allocationId)
+ .withSlotSharingGroupId(sharingGroupId)
+ .withPhysicalSlotResourceProfile(physicalSlotResourceProfile)
+ .withTaskResourceProfile(taskResourceProfile)
+ .build()
+ );
+
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+ assertThat(slotProvider.getSlotRequests(), hasSize(1));
+
+ final SlotProfile requestSlotProfile = slotProvider.getSlotRequests().iterator().next().getSlotProfile();
+
+ assertThat(requestSlotProfile.getPreferredAllocations(), contains(allocationId));
+ assertThat(requestSlotProfile.getPreviousExecutionGraphAllocations(), contains(allocationId));
+ assertThat(requestSlotProfile.getTaskResourceProfile(), equalTo(taskResourceProfile));
+ assertThat(requestSlotProfile.getPreferredLocations(), contains(taskManagerLocation));
+ // task resource profile is used instead of slot sharing group resource profile since slot sharing is ignored
+ assertThat(requestSlotProfile.getPhysicalSlotResourceProfile(), equalTo(taskResourceProfile));
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testCoLocationConstraintThrowsException() {
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final CoLocationConstraint coLocationConstraint = new CoLocationGroup().getLocationConstraint(0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Arrays.asList(
Review comment:
```suggestion
final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Collections.singletonList(
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429755150
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequestBulkTracker.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Tracks physical slot request bulks.
+ */
+class PhysicalSlotRequestBulkTracker {
+
+ private final Clock clock;
+
+ /** Timestamps indicate when bulks become unfulfillable. */
+ private final Map<PhysicalSlotRequestBulk, Long> bulkUnfulfillableTimestamps;
+
+ PhysicalSlotRequestBulkTracker(final Clock clock) {
+ this.clock = clock;
+ this.bulkUnfulfillableTimestamps = new IdentityHashMap<>();
+ }
+
+ void track(final PhysicalSlotRequestBulk bulk) {
+ // a bulk is initially unfulfillable
+ bulkUnfulfillableTimestamps.put(bulk, clock.relativeTimeMillis());
+ }
+
+ void untrack(final PhysicalSlotRequestBulk bulk) {
+ bulkUnfulfillableTimestamps.remove(bulk);
+ }
+
+ boolean isTracked(final PhysicalSlotRequestBulk bulk) {
+ return bulkUnfulfillableTimestamps.containsKey(bulk);
+ }
+
+ void markBulkFulfillable(final PhysicalSlotRequestBulk bulk) {
+ checkState(isTracked(bulk));
+
+ bulkUnfulfillableTimestamps.put(bulk, Long.MAX_VALUE);
+ }
+
+ void markBulkUnfulfillable(final PhysicalSlotRequestBulk bulk, final long currentTimestamp) {
Review comment:
This is to reduce the invocations of `System.nanoTime()` and ensures the timestamp is consistent in `SchedulerImpl#checkPhysicalSlotRequestBulkTimeout(...)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5607dc27dad87a19fe195670b6e216bc2850cb2e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180)
* 9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429758061
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequest.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+
+/**
+ * Represents a request for a physical slot.
+ */
+public class PhysicalSlotRequest {
+
+ private SlotRequestId slotRequestId;
Review comment:
ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-643205521
I just found a problem that in `DefaultExecutionSlotAllocator` the slot request canceler was registered with an incorrect slot request id. It is fixed now and the diff can be found at https://github.com/zhuzhurk/flink/commit/45e5e7f306e5054ff0905879696accb6f1c282ef.
@azagrebin would you take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
}, {
"hash" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401)
* 1253ab0db340f4d3aac238a3c2bc8f8f4abb6941 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] GJL commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429309837
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequestBulk.java
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Represents a bulk of physical slot requests.
+ */
+public class PhysicalSlotRequestBulk {
+
+ final Map<SlotRequestId, PhysicalSlotRequest> pendingRequests;
Review comment:
`private`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk removed a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk removed a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-643052813
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
}, {
"hash" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530",
"triggerID" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"triggerType" : "PUSH"
}, {
"hash" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 1253ab0db340f4d3aac238a3c2bc8f8f4abb6941 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530)
* a637f6415f4f46bc9cc576e0a170cab37266c17a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] GJL commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429217557
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequest.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+
+/**
+ * Represents a request for a physical slot.
+ */
+public class PhysicalSlotRequest {
+
+ private SlotRequestId slotRequestId;
+
+ private SlotProfile slotProfile;
+
+ private boolean slotWillBeOccupiedIndefinitely;
+
+ public PhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile,
+ final boolean slotWillBeOccupiedIndefinitely) {
+
+ this.slotRequestId = slotRequestId;
+ this.slotProfile = slotProfile;
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ }
+
+ public SlotRequestId getSlotRequestId() {
+ return slotRequestId;
+ }
+
+ public SlotProfile getSlotProfile() {
+ return slotProfile;
+ }
+
+ boolean willSlotBeOccupiedIndefinitely() {
+ return slotWillBeOccupiedIndefinitely;
+ }
+
+ /**
+ * Result of a {@link PhysicalSlotRequest}.
+ */
+ public static class Result {
+
+ private SlotRequestId slotRequestId;
Review comment:
Should be `final`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequest.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+
+/**
+ * Represents a request for a physical slot.
+ */
+public class PhysicalSlotRequest {
+
+ private SlotRequestId slotRequestId;
Review comment:
Should be `final`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -67,23 +76,211 @@
private final SlotProviderStrategy slotProviderStrategy;
+ private final SlotOwner slotOwner;
+
private final InputsLocationsRetriever inputsLocationsRetriever;
+ // temporary hack. can be removed along with the individual slot allocation code path
+ // once bulk slot allocation is fully functional
+ private final boolean enableBulkSlotAllocation;
Review comment:
Why not use Javadoc style comments?
Furthermore, I would add more details:
1. What does this flag do?
1. Why is it a temporary hack?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -67,23 +76,211 @@
private final SlotProviderStrategy slotProviderStrategy;
+ private final SlotOwner slotOwner;
+
private final InputsLocationsRetriever inputsLocationsRetriever;
+ // temporary hack. can be removed along with the individual slot allocation code path
+ // once bulk slot allocation is fully functional
+ private final boolean enableBulkSlotAllocation;
Review comment:
Would it be possible to move the new logic to a new class (e.g., `OneSlotPerExecutionSlotAllocator`) that subclasses from `DefaultExecutionSlotAllocator`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SingleLogicalSlot.java
##########
@@ -168,6 +185,11 @@ public void release(Throwable cause) {
releaseFuture.complete(null);
}
+ @Override
+ public boolean willOccupySlotIndefinitely() {
Review comment:
Method name doesn't match the field name `willOccupySlotIndefinitely` vs `willBeOccupiedIndefinitely`. Is it intentional?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequestBulkTracker.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Tracks physical slot request bulks.
+ */
+class PhysicalSlotRequestBulkTracker {
+
+ private final Clock clock;
+
+ /** Timestamps indicate when bulks become unfulfillable. */
+ private final Map<PhysicalSlotRequestBulk, Long> bulkUnfulfillableTimestamps;
+
+ PhysicalSlotRequestBulkTracker(final Clock clock) {
+ this.clock = clock;
+ this.bulkUnfulfillableTimestamps = new IdentityHashMap<>();
+ }
+
+ void track(final PhysicalSlotRequestBulk bulk) {
+ // a bulk is initially unfulfillable
+ bulkUnfulfillableTimestamps.put(bulk, clock.relativeTimeMillis());
+ }
+
+ void untrack(final PhysicalSlotRequestBulk bulk) {
+ bulkUnfulfillableTimestamps.remove(bulk);
+ }
+
+ boolean isTracked(final PhysicalSlotRequestBulk bulk) {
+ return bulkUnfulfillableTimestamps.containsKey(bulk);
+ }
+
+ void markBulkFulfillable(final PhysicalSlotRequestBulk bulk) {
+ checkState(isTracked(bulk));
+
+ bulkUnfulfillableTimestamps.put(bulk, Long.MAX_VALUE);
+ }
+
+ void markBulkUnfulfillable(final PhysicalSlotRequestBulk bulk, final long currentTimestamp) {
Review comment:
What is the benefit of passing an external timestamp instead of calling `clock.relativeTimeMillis()` inside this method?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -67,23 +76,211 @@
private final SlotProviderStrategy slotProviderStrategy;
+ private final SlotOwner slotOwner;
+
private final InputsLocationsRetriever inputsLocationsRetriever;
+ // temporary hack. can be removed along with the individual slot allocation code path
+ // once bulk slot allocation is fully functional
+ private final boolean enableBulkSlotAllocation;
+
public DefaultExecutionSlotAllocator(
SlotProviderStrategy slotProviderStrategy,
- InputsLocationsRetriever inputsLocationsRetriever) {
+ InputsLocationsRetriever inputsLocationsRetriever,
+ boolean enableBulkSlotAllocation) {
this.slotProviderStrategy = checkNotNull(slotProviderStrategy);
this.inputsLocationsRetriever = checkNotNull(inputsLocationsRetriever);
+ this.enableBulkSlotAllocation = enableBulkSlotAllocation;
+
+ this.slotOwner = new DefaultExecutionSlotAllocatorSlotOwner();
pendingSlotAssignments = new HashMap<>();
}
+ /**
+ * Allocate slots in bulks. It has some limitations at the moment and is for pipelined region scheduling only.
+ * Slot sharing will be ignored.
+ * Co-location constraints are not allowed as a result.
+ * Intra-bulk input location preferences will be ignored.
+ */
+ private List<SlotExecutionVertexAssignment> allocateSlotsInBulks(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
+ }
+ }
+
@Override
public List<SlotExecutionVertexAssignment> allocateSlotsFor(
Review comment:
I would prefer this method (`allocateSlotsFor()`) to be the first in the class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 7d4734369b4b87997cce4f593cfd706b78b212f6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r437991598
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
Review comment:
Are you worried about performance for `new ArrayList<>(slotExecutionVertexAssignments.values())` to make `slotExecutionVertexAssignments` a map?
Then you would not need `allExecutionVertexIds` in `createPhysicalSlotRequestFutures` as it could be just `slotExecutionVertexAssignments.keySet()`.
Also not sure, why the API is to return `List<SlotExecutionVertexAssignment>`. Do we need list indexing anywhere on the using side? Otherwise, `allocateSlotsFor` could return `Collection`, then it could return just `slotExecutionVertexAssignments.values()`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436239625
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
Review comment:
Good suggestion!
I had wanted but failed to do it due to the `slotRequestFailureHandler` parameter of `createAndRegisterSlotExecutionVertexAssignment` was not well designed so that slotRequestId was needed before creating `SlotExecutionVertexAssignment`s. Just changed it in b0368079607316f7bc0791609c7b2128750740c3.
This method is now reorganized regarding your suggestions with some small differences:
1. added `slotRequestId` to `SlotExecutionVertexAssignment` and removed `generateExecutionVertexSlotRequestIds `
2. `allExecutionVertexIds` is moved into `createPhysicalSlotRequestFutures(...)`
3. `allocateSlotsForAssignments` is simplified
4. statements are reordered
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438859474
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -77,22 +75,20 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
- executionVertexId,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
+ final CompletableFuture<SlotProfile> slotProfileFuture = getSlotProfileFuture(
+ schedulingRequirements,
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ Collections.emptySet(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile -> slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
Review comment:
Yes it's good to have. done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5607dc27dad87a19fe195670b6e216bc2850cb2e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180)
* 9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r431983175
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
+
+ final CompletableFuture<PhysicalSlot> slotFuture;
+ if (availablePhysicalSlot.isPresent()) {
+ slotFuture = CompletableFuture.completedFuture(availablePhysicalSlot.get().getSlot());
+ } else if (physicalSlotRequest.willSlotBeOccupiedIndefinitely()) {
+ slotFuture = slotPool.requestNewAllocatedSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile(), timeout);
+ } else {
+ slotFuture = slotPool.requestNewAllocatedBatchSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile());
Review comment:
done via a758c3a88a0a6cc2e0b183b20a17fe618f8cbbb9
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
}, {
"hash" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530",
"triggerID" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"triggerType" : "PUSH"
}, {
"hash" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3549",
"triggerID" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 1253ab0db340f4d3aac238a3c2bc8f8f4abb6941 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530)
* a637f6415f4f46bc9cc576e0a170cab37266c17a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3549)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
}, {
"hash" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530",
"triggerID" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 1253ab0db340f4d3aac238a3c2bc8f8f4abb6941 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 41bc7222db9beee142d3b0e80f51ec8fd80c60a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320)
* a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 265fa7767926382824b2894b2662d3a93a76b176 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885)
* edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429777322
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -67,23 +76,211 @@
private final SlotProviderStrategy slotProviderStrategy;
+ private final SlotOwner slotOwner;
+
private final InputsLocationsRetriever inputsLocationsRetriever;
+ // temporary hack. can be removed along with the individual slot allocation code path
+ // once bulk slot allocation is fully functional
+ private final boolean enableBulkSlotAllocation;
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cce8477d3eae8bad8b4109fa1f85e810220b6d39 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117)
* 7f16b12b9c89b723a6c201da252d65e259a53c84 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438082842
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
Review comment:
`createPhysicalSlotRequestFutures ()` does not take `slotExecutionVertexAssignments`
as a param. However, we can use the keySet of `executionVertexSlotRequestIds` as `allExecutionVertexIds`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436239010
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -189,7 +192,7 @@ private DefaultExecutionSlotAllocator createExecutionSlotAllocator(InputsLocatio
return schedulingRequirements;
}
- private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
+ static SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
Review comment:
Agreed. Added util classes in 216a75a34f2d090d726673dc272ea2801fa6eef8 and 277ee720dee5ab63dade29c7178d9e7af69a3f21
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916)
* cce8477d3eae8bad8b4109fa1f85e810220b6d39 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
}, {
"hash" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530",
"triggerID" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"triggerType" : "PUSH"
}, {
"hash" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3549",
"triggerID" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"triggerType" : "PUSH"
}, {
"hash" : "3e1f4485f91ab4c5537c5b6f3170391f25258f9a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3559",
"triggerID" : "3e1f4485f91ab4c5537c5b6f3170391f25258f9a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 1253ab0db340f4d3aac238a3c2bc8f8f4abb6941 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530)
* a637f6415f4f46bc9cc576e0a170cab37266c17a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3549)
* 3e1f4485f91ab4c5537c5b6f3170391f25258f9a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3559)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12256: [FLINK-17018][runtime][WIP] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631012824
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 265fa7767926382824b2894b2662d3a93a76b176 (Tue May 19 18:51:00 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-17018).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r440286226
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator implements SlotOwner {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
Review comment:
Just realized you proposed `LinkedHashMap`. It should work. done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r440243611
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator implements SlotOwner {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
Review comment:
The main problem is that the list order will not match the given order if we use `new ArrayList<>(slotExecutionVertexAssignments.values())`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 41bc7222db9beee142d3b0e80f51ec8fd80c60a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r440199756
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator implements SlotOwner {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
Review comment:
Thanks for the fix @zhuzhurk
I am wondering whether we need `slotRequestId` in `SlotExecutionVertexAssignment`.
If we make `slotExecutionVertexAssignments` `LinkedHashMap<SlotRequestId, SlotExecutionVertexAssignment>` and return `new ArrayList<>(slotExecutionVertexAssignments.values())`, we also do not need `requestToAssignmentMapping` in `allocateSlotsForAssignments` as it can be `slotExecutionVertexAssignments.ketSet()`.
In tests, we could capture `SlotRequestId` from `slotProvider.getSlotRequests/getSlotAllocationRequests`.
Does it makes sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime][WIP] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 265fa7767926382824b2894b2662d3a93a76b176 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r431020373
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
Review comment:
Ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184)
* f009698034f1c67e698a83e43bbb2bcc160f264d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429753738
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SingleLogicalSlot.java
##########
@@ -168,6 +185,11 @@ public void release(Throwable cause) {
releaseFuture.complete(null);
}
+ @Override
+ public boolean willOccupySlotIndefinitely() {
Review comment:
It is intentional.
SingleLogicalSlot has 2 roles. One is logical slot and the other is physical slot payload.
It will occupy a physical slot indefinitely iff it will be occupied indefinitely by a task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eba9c31264c2a0fe9f5bf955ba21de081bd1aa25 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858)
* 5607dc27dad87a19fe195670b6e216bc2850cb2e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180)
* 9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r431939566
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java
##########
@@ -469,6 +469,47 @@ public void releaseSlot(@Nonnull SlotRequestId slotRequestId, @Nullable Throwabl
.collect(Collectors.toList());
}
+ @Override
+ public boolean isSlotRequestBulkFulfillable(final PhysicalSlotRequestBulk slotRequestBulk) {
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r431940611
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r430231588
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
I think that the majority part of `SchedulerImpl` will not be needed anymore in the future.
Introducing a separate interface like `BulkSlotProvider ` might make it easier for us to drop the deprecated components in the future.
> Do you think we can reuse SchedulerImpl in future?
I think not.
> Would just duplicating tryAllocateFromAvailable/cancelSlotRequest in BulkSlotProviderImpl bring less confusion in future?
I think yes. Let's do it this way.
> Will we need single slot provider eventually for pipeline region scheduling at all?
I think not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f181cf943442db6173eba4c817f40bbab49538da Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857)
* eba9c31264c2a0fe9f5bf955ba21de081bd1aa25 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438082842
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
Review comment:
`createPhysicalSlotRequestFutures ()` does not take `slotExecutionVertexAssignments`
as a param. However, we can use the keySet of `executionVertexSlotRequestIds` as `allExecutionVertexIds`. Done via 9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432270529
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
Review comment:
done via 2c5a1da428b68247e42c9e51e30bbbec278e40e8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436060993
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotWillBeOccupiedIndefinitely);
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProvider.allocatePhysicalSlots(physicalSlotRequests, allocationTimeout))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
Review comment:
Sure, that's what we had wanted when adding `SingleLogicalSlot#createAndAllocateFromPhysicalSlot`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ba2a2eb65072f71c66055c3f86549a061eb1514 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184)
* f009698034f1c67e698a83e43bbb2bcc160f264d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227)
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2aa39a469f86ceae75eafb75194f7c7f82e3a6cc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429754093
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -67,23 +76,211 @@
private final SlotProviderStrategy slotProviderStrategy;
+ private final SlotOwner slotOwner;
+
private final InputsLocationsRetriever inputsLocationsRetriever;
+ // temporary hack. can be removed along with the individual slot allocation code path
+ // once bulk slot allocation is fully functional
+ private final boolean enableBulkSlotAllocation;
Review comment:
Good suggestion! Will give a try.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436238884
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotWillBeOccupiedIndefinitely);
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProvider.allocatePhysicalSlots(physicalSlotRequests, allocationTimeout))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r433106488
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
PR #12423 is opened which is based on this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432385225
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
FLINK-18034 is opened to "Introduce PreferredLocationsRetriever".
With it we can go one step further to centrally managed the preferred location calculation regardless of state-based or inputs-based.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438058712
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements);
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = slotExecutionVertexAssignments
+ .stream()
+ .collect(
+ Collectors.toMap(
+ SlotExecutionVertexAssignment::getExecutionVertexId,
+ SlotExecutionVertexAssignment::getSlotRequestId));
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments);
+
+ return Collections.unmodifiableList(slotExecutionVertexAssignments);
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ (slotRequestId, throwable) -> slotProvider.cancelSlotRequest(slotRequestId, throwable));
Review comment:
done.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/ExecutionSlotAllocatorTestUtils.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Test utils for {@link ExecutionSlotAllocator}.
+ */
+public class ExecutionSlotAllocatorTestUtils {
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-635178001
@azagrebin @GJL #12375 is opened for bulk slot allocation part with a `BulkSlotProvider` introduced. Related comments above are also addressed in it.
Would you take a look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk merged pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk merged pull request #12256:
URL: https://github.com/apache/flink/pull/12256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429777663
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -67,23 +76,211 @@
private final SlotProviderStrategy slotProviderStrategy;
+ private final SlotOwner slotOwner;
+
private final InputsLocationsRetriever inputsLocationsRetriever;
+ // temporary hack. can be removed along with the individual slot allocation code path
+ // once bulk slot allocation is fully functional
+ private final boolean enableBulkSlotAllocation;
+
public DefaultExecutionSlotAllocator(
SlotProviderStrategy slotProviderStrategy,
- InputsLocationsRetriever inputsLocationsRetriever) {
+ InputsLocationsRetriever inputsLocationsRetriever,
+ boolean enableBulkSlotAllocation) {
this.slotProviderStrategy = checkNotNull(slotProviderStrategy);
this.inputsLocationsRetriever = checkNotNull(inputsLocationsRetriever);
+ this.enableBulkSlotAllocation = enableBulkSlotAllocation;
+
+ this.slotOwner = new DefaultExecutionSlotAllocatorSlotOwner();
pendingSlotAssignments = new HashMap<>();
}
+ /**
+ * Allocate slots in bulks. It has some limitations at the moment and is for pipelined region scheduling only.
+ * Slot sharing will be ignored.
+ * Co-location constraints are not allowed as a result.
+ * Intra-bulk input location preferences will be ignored.
+ */
+ private List<SlotExecutionVertexAssignment> allocateSlotsInBulks(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
+ }
+ }
+
@Override
public List<SlotExecutionVertexAssignment> allocateSlotsFor(
Review comment:
Done with methods of `OneSlotPerExecutionSlotAllocator` moved out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438859991
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/AbstractExecutionSlotAllocator.java
##########
@@ -94,7 +96,26 @@ protected SlotExecutionVertexAssignment createAndRegisterSlotExecutionVertexAssi
return slotExecutionVertexAssignment;
}
- protected CompletableFuture<Collection<TaskManagerLocation>> calculatePreferredLocations(
+ protected CompletableFuture<SlotProfile> getSlotProfileFuture(
+ final ExecutionVertexSchedulingRequirements schedulingRequirements,
+ final ResourceProfile physicalSlotResourceProfile,
+ final Set<ExecutionVertexID> producersToIgnore,
+ final Set<AllocationID> allPreviousAllocationIds) {
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(schedulingRequirements.getExecutionVertexId(), producersToIgnore);
+
+ return preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ physicalSlotResourceProfile,
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+ }
+
+ private CompletableFuture<Collection<TaskManagerLocation>> calculatePreferredLocations(
Review comment:
true. removed.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/AbstractExecutionSlotAllocator.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiConsumer;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Base class for all {@link ExecutionSlotAllocator}. It is responsible to allocate slots for tasks and
+ * keep the unfulfilled slot requests for further cancellation.
+ */
+abstract class AbstractExecutionSlotAllocator implements ExecutionSlotAllocator {
+
+ private final Map<ExecutionVertexID, SlotExecutionVertexAssignment> pendingSlotAssignments;
+
+ private final PreferredLocationsRetriever preferredLocationsRetriever;
+
+ protected AbstractExecutionSlotAllocator(final PreferredLocationsRetriever preferredLocationsRetriever) {
+ this.preferredLocationsRetriever = checkNotNull(preferredLocationsRetriever);
+ this.pendingSlotAssignments = new HashMap<>();
+ }
+
+ @Override
+ public void cancel(final ExecutionVertexID executionVertexId) {
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment = pendingSlotAssignments.get(executionVertexId);
+ if (slotExecutionVertexAssignment != null) {
+ slotExecutionVertexAssignment.getLogicalSlotFuture().cancel(false);
+ }
+ }
+
+ protected void validateSchedulingRequirements(
Review comment:
Sure. We can make it package private for now given that we do not expect it to be implemented somewhere out of the package. The same applies for the other protected methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432347996
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
In the latest update, I removed `createSlotProfile()` because I find it does not reduce much code duplication but introduced a nested method with weird parameters to distinguish `DefaultExecutionSlotAllocator` and `OneSlotPerExecutionSlotAllocator `.
I think the major part that would be good to extract out from `ExecutionSlotAllocator` is the preferred location calculation. We can introduce a `PreferredLocationsRetriever` which wraps `InputsLocationsRetriever`, `calculatePreferredLocations(...)` and `getPreferredLocationsBasedOnInputs(...)`.
I think we can open another task to do it since it is not very related to this change. (I'm already working on it though.)
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432385225
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
FLINK-18034 is opened to "Introduce PreferredLocationsRetriever".
With it we can go one step further to centrally manage the preferred location calculation regardless of state-based or inputs-based.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eba9c31264c2a0fe9f5bf955ba21de081bd1aa25 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429801548
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
+
+ @Test
+ public void testSingleTaskOccupyingSlotIndefinitely() {
+ final PhysicalSlot physicalSlot = createPhysicalSlot();
+ allocateSingleLogicalSlotFromPhysicalSlot(physicalSlot, true);
+
+ assertTrue(physicalSlot.willBeOccupiedIndefinitely());
+ }
+
+ @Test
+ public void testSingleTaskNotOccupyingSlotIndefinitely() {
+ final PhysicalSlot physicalSlot = createPhysicalSlot();
+ allocateSingleLogicalSlotFromPhysicalSlot(physicalSlot, true);
+
+ assertTrue(physicalSlot.willBeOccupiedIndefinitely());
Review comment:
```suggestion
allocateSingleLogicalSlotFromPhysicalSlot(physicalSlot, false);
assertFalse(physicalSlot.willBeOccupiedIndefinitely());
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java
##########
@@ -469,6 +469,47 @@ public void releaseSlot(@Nonnull SlotRequestId slotRequestId, @Nullable Throwabl
.collect(Collectors.toList());
}
+ @Override
+ public boolean isSlotRequestBulkFulfillable(final PhysicalSlotRequestBulk slotRequestBulk) {
+ final Set<AllocationID> assignedSlots = new HashSet<>(slotRequestBulk.getFulfilledRequests().values());
+ final Set<SlotInfo> reusableSlots = getReusableSlots(assignedSlots);
+ return areRequestsFulfillableWithSlots(slotRequestBulk.getPendingRequests().values(), reusableSlots);
+ }
+
+ private Set<SlotInfo> getReusableSlots(final Set<AllocationID> slotsToExclude) {
+ return Stream
+ .concat(
+ getAvailableSlotsInformation().stream(),
+ getAllocatedSlotsInformation().stream())
+ .filter(slotInfo -> !slotInfo.willBeOccupiedIndefinitely())
+ .filter(slotInfo -> !slotsToExclude.contains(slotInfo.getAllocationId()))
+ .collect(Collectors.toSet());
+ }
+
+ private static boolean areRequestsFulfillableWithSlots(
+ final Collection<PhysicalSlotRequest> requests,
+ final Set<SlotInfo> slots) {
+
+ final Set<SlotInfo> remainingSlots = new HashSet<>(slots);
+ for (PhysicalSlotRequest request : requests) {
+ final Optional<SlotInfo> matchedSlot = findMatchingSlotForRequest(request, remainingSlots);
+ if (matchedSlot.isPresent()) {
+ remainingSlots.remove(matchedSlot.get());
+ } else {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private static Optional<SlotInfo> findMatchingSlotForRequest(
+ final PhysicalSlotRequest request,
+ final Collection<SlotInfo> slots) {
+
+ final ResourceProfile requiredResource = request.getSlotProfile().getPhysicalSlotResourceProfile();
Review comment:
Do we need to keep `PhysicalSlotRequests` in `PhysicalSlotRequestBulk`?
It seems we use only ids and `ResourceProfile` of `PhysicalSlotRequest.SlotProfile`.
Would it be enough for now to keep only `ResourceProfiles` in `PhysicalSlotRequestBulk.pendingRequests`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
Could we introduce `new SlotProfileRetriever(inputsLocationsRetriever, ignoreInternalProducers, useTaskProfile).createSlotProfileFutures(Collection<ExecutionVertexSchedulingRequirements>)`?
This would deduplicate a lot of code in `OneSlotPerExecutionSlotAllocator/DefaultExecutionSlotAllocator`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
+
+ final CompletableFuture<PhysicalSlot> slotFuture;
+ if (availablePhysicalSlot.isPresent()) {
+ slotFuture = CompletableFuture.completedFuture(availablePhysicalSlot.get().getSlot());
+ } else if (physicalSlotRequest.willSlotBeOccupiedIndefinitely()) {
+ slotFuture = slotPool.requestNewAllocatedSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile(), timeout);
+ } else {
+ slotFuture = slotPool.requestNewAllocatedBatchSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile());
Review comment:
These requests to `SlotPoolImpl` will also schedule per-slot timeout in streaming and additional fullfilability check for batch slot in `SlotPoolImpl`. Do we want to keep this complication, especially in future? Bulk timeout seems to be enough and basically addresses the same.
Would it be easier to add e.g. `SlotPoolImpl#requestNewAllocatedSlotWithoutTimeout`? `requestNewAllocatedSlotWithoutTimeout` could be similar to `requestNewAllocatedSlot` for streaming but w/o any per-slot timeout tracking in `SlotPoolImpl`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
It looks like `componentMainThreadExecutor/tryAllocateFromAvailable/cancelSlotRequest` are the only things, we reuse in `SchedulerImpl`.
Not sure whether it makes sense to extend this class which has not been designed for bulk slot allocation.
Do you think we can reuse `SchedulerImpl` in future?
Could we just introduce another component for the change in this commit (basically bulk timeout), e.g. `BulkSlotProvider.allocatePhysicalSlots/cancelSlotRequest` interface or so?
Would just duplicating `tryAllocateFromAvailable/cancelSlotRequest` in `BulkSlotProviderImpl` bring less confusion in future?
Will we need single slot provider eventually for pipeline region scheduling at all?
Later we could see how to introduce slot sharing and co-location concerns, maybe as other components on top.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java
##########
@@ -469,6 +469,47 @@ public void releaseSlot(@Nonnull SlotRequestId slotRequestId, @Nullable Throwabl
.collect(Collectors.toList());
}
+ @Override
+ public boolean isSlotRequestBulkFulfillable(final PhysicalSlotRequestBulk slotRequestBulk) {
Review comment:
Do we need to mix this concern of bulk allocation to `SlotPool`?
Could we keep the `SlotPool` as only single physical slot allocator?
Would it not be more natural to expose `SlotPool#getAllocatedSlotsInformation` and move the bulk logic to its tracker?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/SlotProviderStrategy.java
##########
@@ -56,6 +62,24 @@
ScheduledUnit scheduledUnit,
SlotProfile slotProfile);
+ /**
+ * Allocates a bulk of physical slots.
+ *
+ * @param physicalSlotRequests requests for physical slots
+ * @return future of the allocation which will be completed normally only when all the requests are fulfilled
+ */
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
Review comment:
Do we need `SlotProviderStrategy` in `OneSlotPerExecutionSlotAllocator` at all?
Could we use directly new `BulkSlotProvider.allocatePhysicalSlots/cancelSlotRequest` and create `OneSlotPerExecutionSlotAllocator` with `willSlotBeOccupiedIndefinitely` flag based on `ScheduleMode`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
Review comment:
`OneSlotPerExecutionSlotAllocator` and `DefaultExecutionSlotAllocator` implement different approaches.
Would it make sense to introduce a base `AbstractExecutionSlotAllocator` and inherit it by both classes?
There will be not much in common, mostly some state management around `pendingSlotAssignments`, if we:
- introduce a separate `SlotProfileRetriever`
- use `BulkSlotProvider` and `willSlotBeOccupiedIndefinitely` flag in `OneSlotPerExecutionSlotAllocator` instead of `slotProviderStrategy`
- maybe move static methods to utils if it make sense
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotOwner slotOwner;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProviderStrategy slotProviderStrategy,
+ final InputsLocationsRetriever inputsLocationsRetriever) {
+ super(slotProviderStrategy, inputsLocationsRetriever);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
Review comment:
Should we also release the slot from `SlotPool`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
+
+ final CompletableFuture<PhysicalSlot> slotFuture;
+ if (availablePhysicalSlot.isPresent()) {
+ slotFuture = CompletableFuture.completedFuture(availablePhysicalSlot.get().getSlot());
+ } else if (physicalSlotRequest.willSlotBeOccupiedIndefinitely()) {
+ slotFuture = slotPool.requestNewAllocatedSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile(), timeout);
+ } else {
+ slotFuture = slotPool.requestNewAllocatedBatchSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile());
Review comment:
Yes, this is what I also thought about.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
True, there is no duplication at the moment.
I meant to move the concern of `SlotProfileRetriever` (and all related code) into a separate component to reuse it in `OneSlotPerExecutionSlotAllocator/DefaultExecutionSlotAllocator`. This would simplify `OneSlotPerExecutionSlotAllocator/DefaultExecutionSlotAllocator`, imo, and there will be also no code duplication.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916)
* cce8477d3eae8bad8b4109fa1f85e810220b6d39 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca05fc241f6fd0550a39612b4ddef86b63a304a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625)
* f181cf943442db6173eba4c817f40bbab49538da UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438082842
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
Review comment:
`createPhysicalSlotRequestFutures ()` does not take `slotExecutionVertexAssignments`
as a param.
But you are right that we do not need to compute a `allExecutionVertexIds `. We can directly use the keySet of `executionVertexSlotRequestIds` as `allExecutionVertexIds`. Done via 7d4734369b4b87997cce4f593cfd706b78b212f6.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] GJL commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
GJL commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r430465756
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
Review comment:
```suggestion
public class PhysicalSlotOccupationTest extends TestLogger {
```
so that `Ctrl + Shift + T` works in IDEA: https://www.jetbrains.com/help/idea/navigating-between-test-and-test-subject.html
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
Review comment:
```suggestion
public class AllocatedSlotOccupationTest extends TestLogger {
```
so that `Ctrl + Shift + T` works in IDEA: https://www.jetbrains.com/help/idea/navigating-between-test-and-test-subject.html
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
Review comment:
What is the subject under test here? I assume `AllocatedSlot`.
It would be nice to rename this class `AllocatedSlotOccupationTest` so that `Ctrl + Shift +T` works in IDEA: https://www.jetbrains.com/help/idea/navigating-between-test-and-test-subject.html
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
Review comment:
What is the subject under test here? I assume `AllocatedSlot`. It would be nice to rename this class `AllocatedSlotOccupationTest` so that `Ctrl + Shift +T` works in IDEA: https://www.jetbrains.com/help/idea/navigating-between-test-and-test-subject.html
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
Review comment:
What is the subject under test here? I assume `AllocatedSlot`. It would be nice to rename this class `AllocatedSlotOccupationTest` so that `Ctrl + Shift + T` works in IDEA: https://www.jetbrains.com/help/idea/navigating-between-test-and-test-subject.html
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotOwner slotOwner;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProviderStrategy slotProviderStrategy,
+ final InputsLocationsRetriever inputsLocationsRetriever) {
+ super(slotProviderStrategy, inputsLocationsRetriever);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
Review comment:
Under what circumstances can it happen that the physical slot is already assigned a payload?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438858533
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocatorTest.java
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraint;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroup;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotTestUtils.createPhysicalSlot;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.createSchedulingRequirements;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.findSlotAssignmentByExecutionVertexId;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for {@link OneSlotPerExecutionSlotAllocator}.
+ */
+public class OneSlotPerExecutionSlotAllocatorTest extends TestLogger {
+
+ private TestingBulkSlotProvider slotProvider;
+
+ @Before
+ public void setUp() throws Exception {
+ slotProvider = new TestingBulkSlotProvider();
+ }
+
+ @Test
+ public void testSucceededSlotAllocation() {
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(1));
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getExecutionVertexId(), equalTo(executionVertexID));
+ assertThat(slotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(false));
+ }
+
+ @Test
+ public void testFailedSlotAllocation() {
+ final OneSlotPerExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ slotProvider.forceFailingSlotAllocation();
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(true));
+ assertThat(executionSlotAllocator.getPendingSlotAssignments().keySet(), hasSize(0));
+ assertThat(slotProvider.getCancelledSlotRequestIds(), contains(slotAssignment.getSlotRequestId()));
+ }
+
+ @Test
+ public void testInterBulkInputLocationPreferencesAreRespected() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForProducer =
+ createSchedulingRequirements(producerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForProducer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForProducer);
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignmentsForProducer);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+
+ inputsLocationsRetriever.markScheduled(consumerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForConsumer =
+ createSchedulingRequirements(consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForConsumer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForConsumer);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignmentsForConsumer);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(false));
+
+ inputsLocationsRetriever.assignTaskManagerLocation(producerId);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testIntraBulkInputLocationPreferencesDoNotBlockAllocation() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ inputsLocationsRetriever.markScheduled(consumerId);
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(producerId, consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(2));
+
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignments);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignments);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testCreatedSlotRequests() {
+ final ExecutionVertexID executionVertexId = new ExecutionVertexID(new JobVertexID(), 0);
+ final AllocationID allocationId = new AllocationID();
+ final SlotSharingGroupId sharingGroupId = new SlotSharingGroupId();
+ final ResourceProfile taskResourceProfile = ResourceProfile.fromResources(0.5, 250);
+ final ResourceProfile physicalSlotResourceProfile = ResourceProfile.fromResources(1.0, 300);
+ final TaskManagerLocation taskManagerLocation = new LocalTaskManagerLocation();
+
+ final TestingStateLocationRetriever stateLocationRetriever = new TestingStateLocationRetriever();
+ stateLocationRetriever.setStateLocation(executionVertexId, taskManagerLocation);
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ stateLocationRetriever,
+ new TestingInputsLocationsRetriever.Builder().build());
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Arrays.asList(
+ new ExecutionVertexSchedulingRequirements.Builder()
+ .withExecutionVertexId(executionVertexId)
+ .withPreviousAllocationId(allocationId)
+ .withSlotSharingGroupId(sharingGroupId)
+ .withPhysicalSlotResourceProfile(physicalSlotResourceProfile)
+ .withTaskResourceProfile(taskResourceProfile)
+ .build()
+ );
+
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+ assertThat(slotProvider.getSlotRequests(), hasSize(1));
+
+ final SlotProfile requestSlotProfile = slotProvider.getSlotRequests().iterator().next().getSlotProfile();
+
+ assertThat(requestSlotProfile.getPreferredAllocations(), contains(allocationId));
+ assertThat(requestSlotProfile.getPreviousExecutionGraphAllocations(), contains(allocationId));
+ assertThat(requestSlotProfile.getTaskResourceProfile(), equalTo(taskResourceProfile));
+ assertThat(requestSlotProfile.getPreferredLocations(), contains(taskManagerLocation));
+ // task resource profile is used instead of slot sharing group resource profile since slot sharing is ignored
+ assertThat(requestSlotProfile.getPhysicalSlotResourceProfile(), equalTo(taskResourceProfile));
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testCoLocationConstraintThrowsException() {
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final CoLocationConstraint coLocationConstraint = new CoLocationGroup().getLocationConstraint(0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Arrays.asList(
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r435061970
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
Review comment:
Could we already make `OneSlotPerExecutionSlotAllocator` depending on `BulkSlotProvider` if `Scheduler` extends `BulkSlotProvider`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429754093
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -67,23 +76,211 @@
private final SlotProviderStrategy slotProviderStrategy;
+ private final SlotOwner slotOwner;
+
private final InputsLocationsRetriever inputsLocationsRetriever;
+ // temporary hack. can be removed along with the individual slot allocation code path
+ // once bulk slot allocation is fully functional
+ private final boolean enableBulkSlotAllocation;
Review comment:
Will give a try.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2aa39a469f86ceae75eafb75194f7c7f82e3a6cc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422)
* 8ba2a2eb65072f71c66055c3f86549a061eb1514 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca05fc241f6fd0550a39612b4ddef86b63a304a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625)
* f181cf943442db6173eba4c817f40bbab49538da Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857)
* eba9c31264c2a0fe9f5bf955ba21de081bd1aa25 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk removed a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk removed a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-643007617
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk removed a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk removed a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-644476328
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
}, {
"hash" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530",
"triggerID" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"triggerType" : "PUSH"
}, {
"hash" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3549",
"triggerID" : "a637f6415f4f46bc9cc576e0a170cab37266c17a",
"triggerType" : "PUSH"
}, {
"hash" : "3e1f4485f91ab4c5537c5b6f3170391f25258f9a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e1f4485f91ab4c5537c5b6f3170391f25258f9a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 1253ab0db340f4d3aac238a3c2bc8f8f4abb6941 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530)
* a637f6415f4f46bc9cc576e0a170cab37266c17a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3549)
* 3e1f4485f91ab4c5537c5b6f3170391f25258f9a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432385225
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
FLINK-18034 is opened to "Introduce PreferredLocationsRetriever".
With it we can go one step further to centrally manage the preferred location calculation regardless of it's state-based or inputs-based.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916)
* cce8477d3eae8bad8b4109fa1f85e810220b6d39 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117)
* 7f16b12b9c89b723a6c201da252d65e259a53c84 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca05fc241f6fd0550a39612b4ddef86b63a304a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-633867170
Thanks for the comments! @azagrebin
Agreed that we can make new allocator and bulk slot allocation independent from existing `SchedulerImp` and `DefaultExecutionSlotAllocator ` for further evolving. Will do it.
>> Also, maybe we could split the PR somehow into e.g. two PRs: bulk slot allocation and OneSlotPerExecutionSlotAllocator to be able to merge smaller chunks of code separately
Sure we can do it. The PR is already in smaller pieces and it would be easy to split the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-644640968
The failed case `StreamingKafkaITCase` is unrelated. The problem is FLINK-18311.
Merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f009698034f1c67e698a83e43bbb2bcc160f264d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227)
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 7d4734369b4b87997cce4f593cfd706b78b212f6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436238809
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
Review comment:
It's done to make `OneSlotPerExecutionSlotAllocator` depend on `BulkSlotProvider`.
I think `cancelSlotRequest` may be better because `SlotPoolImpl#releaseSlot()` also cancels pending physical slot request.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotWillBeOccupiedIndefinitely);
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProvider.allocatePhysicalSlots(physicalSlotRequests, allocationTimeout))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
Review comment:
done.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r431653239
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
+
+ @Test
+ public void testSingleTaskOccupyingSlotIndefinitely() {
+ final PhysicalSlot physicalSlot = createPhysicalSlot();
+ allocateSingleLogicalSlotFromPhysicalSlot(physicalSlot, true);
+
+ assertTrue(physicalSlot.willBeOccupiedIndefinitely());
+ }
+
+ @Test
+ public void testSingleTaskNotOccupyingSlotIndefinitely() {
+ final PhysicalSlot physicalSlot = createPhysicalSlot();
+ allocateSingleLogicalSlotFromPhysicalSlot(physicalSlot, true);
+
+ assertTrue(physicalSlot.willBeOccupiedIndefinitely());
Review comment:
done.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java
##########
@@ -469,6 +469,47 @@ public void releaseSlot(@Nonnull SlotRequestId slotRequestId, @Nullable Throwabl
.collect(Collectors.toList());
}
+ @Override
+ public boolean isSlotRequestBulkFulfillable(final PhysicalSlotRequestBulk slotRequestBulk) {
+ final Set<AllocationID> assignedSlots = new HashSet<>(slotRequestBulk.getFulfilledRequests().values());
+ final Set<SlotInfo> reusableSlots = getReusableSlots(assignedSlots);
+ return areRequestsFulfillableWithSlots(slotRequestBulk.getPendingRequests().values(), reusableSlots);
+ }
+
+ private Set<SlotInfo> getReusableSlots(final Set<AllocationID> slotsToExclude) {
+ return Stream
+ .concat(
+ getAvailableSlotsInformation().stream(),
+ getAllocatedSlotsInformation().stream())
+ .filter(slotInfo -> !slotInfo.willBeOccupiedIndefinitely())
+ .filter(slotInfo -> !slotsToExclude.contains(slotInfo.getAllocationId()))
+ .collect(Collectors.toSet());
+ }
+
+ private static boolean areRequestsFulfillableWithSlots(
+ final Collection<PhysicalSlotRequest> requests,
+ final Set<SlotInfo> slots) {
+
+ final Set<SlotInfo> remainingSlots = new HashSet<>(slots);
+ for (PhysicalSlotRequest request : requests) {
+ final Optional<SlotInfo> matchedSlot = findMatchingSlotForRequest(request, remainingSlots);
+ if (matchedSlot.isPresent()) {
+ remainingSlots.remove(matchedSlot.get());
+ } else {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private static Optional<SlotInfo> findMatchingSlotForRequest(
+ final PhysicalSlotRequest request,
+ final Collection<SlotInfo> slots) {
+
+ final ResourceProfile requiredResource = request.getSlotProfile().getPhysicalSlotResourceProfile();
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r430190027
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java
##########
@@ -469,6 +469,47 @@ public void releaseSlot(@Nonnull SlotRequestId slotRequestId, @Nullable Throwabl
.collect(Collectors.toList());
}
+ @Override
+ public boolean isSlotRequestBulkFulfillable(final PhysicalSlotRequestBulk slotRequestBulk) {
Review comment:
This sounds good to me.
I had wanted to do it like that but encountered some problems. I do not remember the problems unfortunately.
I will try and see if it is still a problem. Maybe not since the design was adjusted during developing.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotOccupationTest.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmanager.slots.TestingSlotOwner;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests whether the slot occupation works correctly.
+ */
+public class SlotOccupationTest extends TestLogger {
+
+ @Test
+ public void testSingleTaskOccupyingSlotIndefinitely() {
+ final PhysicalSlot physicalSlot = createPhysicalSlot();
+ allocateSingleLogicalSlotFromPhysicalSlot(physicalSlot, true);
+
+ assertTrue(physicalSlot.willBeOccupiedIndefinitely());
+ }
+
+ @Test
+ public void testSingleTaskNotOccupyingSlotIndefinitely() {
+ final PhysicalSlot physicalSlot = createPhysicalSlot();
+ allocateSingleLogicalSlotFromPhysicalSlot(physicalSlot, true);
+
+ assertTrue(physicalSlot.willBeOccupiedIndefinitely());
Review comment:
Nice catch!
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java
##########
@@ -469,6 +469,47 @@ public void releaseSlot(@Nonnull SlotRequestId slotRequestId, @Nullable Throwabl
.collect(Collectors.toList());
}
+ @Override
+ public boolean isSlotRequestBulkFulfillable(final PhysicalSlotRequestBulk slotRequestBulk) {
+ final Set<AllocationID> assignedSlots = new HashSet<>(slotRequestBulk.getFulfilledRequests().values());
+ final Set<SlotInfo> reusableSlots = getReusableSlots(assignedSlots);
+ return areRequestsFulfillableWithSlots(slotRequestBulk.getPendingRequests().values(), reusableSlots);
+ }
+
+ private Set<SlotInfo> getReusableSlots(final Set<AllocationID> slotsToExclude) {
+ return Stream
+ .concat(
+ getAvailableSlotsInformation().stream(),
+ getAllocatedSlotsInformation().stream())
+ .filter(slotInfo -> !slotInfo.willBeOccupiedIndefinitely())
+ .filter(slotInfo -> !slotsToExclude.contains(slotInfo.getAllocationId()))
+ .collect(Collectors.toSet());
+ }
+
+ private static boolean areRequestsFulfillableWithSlots(
+ final Collection<PhysicalSlotRequest> requests,
+ final Set<SlotInfo> slots) {
+
+ final Set<SlotInfo> remainingSlots = new HashSet<>(slots);
+ for (PhysicalSlotRequest request : requests) {
+ final Optional<SlotInfo> matchedSlot = findMatchingSlotForRequest(request, remainingSlots);
+ if (matchedSlot.isPresent()) {
+ remainingSlots.remove(matchedSlot.get());
+ } else {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private static Optional<SlotInfo> findMatchingSlotForRequest(
+ final PhysicalSlotRequest request,
+ final Collection<SlotInfo> slots) {
+
+ final ResourceProfile requiredResource = request.getSlotProfile().getPhysicalSlotResourceProfile();
Review comment:
`ResourceProfiles` are enough.
I think you are right that we should simplify it.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
+
+ final CompletableFuture<PhysicalSlot> slotFuture;
+ if (availablePhysicalSlot.isPresent()) {
+ slotFuture = CompletableFuture.completedFuture(availablePhysicalSlot.get().getSlot());
+ } else if (physicalSlotRequest.willSlotBeOccupiedIndefinitely()) {
+ slotFuture = slotPool.requestNewAllocatedSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile(), timeout);
+ } else {
+ slotFuture = slotPool.requestNewAllocatedBatchSlot(slotRequestId, slotProfile.getPhysicalSlotResourceProfile());
Review comment:
I think it's fine to have a `requestNewAllocatedSlotWithoutTimeout()`.
The question is about batch slot requests which also have individual timeout check. The timeout check for batch requests is independent from the slot requesting invocation. (via a regularly scheduled `checkBatchSlotTimeout`)
How about to disable `checkBatchSlotTimeout` on `requestNewAllocatedSlotWithoutTimeout `? It is a bit hack but reduces complication.
In the future, once we dropped individual slot allocation from slot provider, we can drop these individual timeout check.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/SlotProviderStrategy.java
##########
@@ -56,6 +62,24 @@
ScheduledUnit scheduledUnit,
SlotProfile slotProfile);
+ /**
+ * Allocates a bulk of physical slots.
+ *
+ * @param physicalSlotRequests requests for physical slots
+ * @return future of the allocation which will be completed normally only when all the requests are fulfilled
+ */
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
Review comment:
Nice catch!
It's not needed. I kept it to not change creation of `DefaultrExecutionSlotAllocator`. But given that we now have `OneSlotPerExecutionSlotAllocator`, I think we can get rid of it.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotOwner slotOwner;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProviderStrategy slotProviderStrategy,
+ final InputsLocationsRetriever inputsLocationsRetriever) {
+ super(slotProviderStrategy, inputsLocationsRetriever);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
Review comment:
All pending requested and allocated slots will be released on this exception.
This is because `slotProviderStrategy.cancelSlotRequest()` will be invoked.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotOwner slotOwner;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProviderStrategy slotProviderStrategy,
+ final InputsLocationsRetriever inputsLocationsRetriever) {
+ super(slotProviderStrategy, inputsLocationsRetriever);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
Review comment:
All pending requested and allocated slots will be released on this exception.
This is because `slotProviderStrategy.cancelSlotRequest()` will be invoked for each slot request.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
Review comment:
Actually most of the methods of `DefaultExecutionSlotAllocator` are required for `OneSlotPerExecutionSlotAllocator`.
But I think it's fine to extract the common things into a base class.
I will give a try.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -96,52 +97,88 @@ public DefaultExecutionSlotAllocator(
LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
- CompletableFuture<LogicalSlot> slotFuture = calculatePreferredLocations(
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ Collections.emptySet(),
+ schedulingRequirements.getPhysicalSlotResourceProfile(),
+ allPreviousAllocationIds);
+
+ final CompletableFuture<LogicalSlot> slotFuture = slotProfileFuture.thenCompose(
+ slotProfile ->
+ slotProviderStrategy.allocateSlot(
+ slotRequestId,
+ new ScheduledUnit(
+ executionVertexId,
+ slotSharingGroupId,
+ schedulingRequirements.getCoLocationConstraint()),
+ slotProfile));
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
executionVertexId,
- schedulingRequirements.getPreferredLocations(),
- inputsLocationsRetriever,
- Collections.emptySet()).thenCompose(
- (Collection<TaskManagerLocation> preferredLocations) ->
- slotProviderStrategy.allocateSlot(
- slotRequestId,
- new ScheduledUnit(
- executionVertexId,
- slotSharingGroupId,
- schedulingRequirements.getCoLocationConstraint()),
- SlotProfile.priorAllocation(
- schedulingRequirements.getTaskResourceProfile(),
- schedulingRequirements.getPhysicalSlotResourceProfile(),
- preferredLocations,
- Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
- allPreviousAllocationIds)));
-
- SlotExecutionVertexAssignment slotExecutionVertexAssignment =
- new SlotExecutionVertexAssignment(executionVertexId, slotFuture);
- // add to map first to avoid the future completed before added.
- pendingSlotAssignments.put(executionVertexId, slotExecutionVertexAssignment);
-
- slotFuture.whenComplete(
- (ignored, throwable) -> {
- pendingSlotAssignments.remove(executionVertexId);
- if (throwable != null) {
- slotProviderStrategy.cancelSlotRequest(slotRequestId, slotSharingGroupId, throwable);
- }
- });
+ slotFuture,
+ slotRequestId,
+ slotSharingGroupId);
slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
}
return slotExecutionVertexAssignments;
}
- private void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+ protected void validateSchedulingRequirements(Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
schedulingRequirements.stream()
.map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
.forEach(id -> checkState(
!pendingSlotAssignments.containsKey(id),
"BUG: vertex %s tries to allocate a slot when its previous slot request is still pending", id));
}
+ protected CompletableFuture<SlotProfile> createSlotProfile(
Review comment:
This is a shared method. Not sure what is the duplication?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
I think that the majority part of `SchedulerImpl` will be needed anymore in the future.
Introducing a separate interface like `BulkSlotProvider ` might make it easier for us to drop the deprecated components in the future.
> Do you think we can reuse SchedulerImpl in future?
I think not.
> Would just duplicating tryAllocateFromAvailable/cancelSlotRequest in BulkSlotProviderImpl bring less confusion in future?
I think yes. Let's do it this way.
> Will we need single slot provider eventually for pipeline region scheduling at all?
I think not.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
I think that the majority part of `SchedulerImpl` will be needed anymore in the future.
Introducing a separate interface like `BulkSlotProvider ` might make it easier for us to drop the deprecated components in the future.
> Do you think we can reuse SchedulerImpl in future?
I think not.
> Would just duplicating tryAllocateFromAvailable/cancelSlotRequest in BulkSlotProviderImpl bring less confusion in future?
I think yes. Let's do it this way.
> Will we need single slot provider eventually for pipeline region scheduling at all?
I think not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r435062282
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
Review comment:
Could we already make `OneSlotPerExecutionSlotAllocator` depending on `BulkSlotProvider` if `SlotProvider` interface extends `BulkSlotProvider` and we duplicate `cancelSlotRequest` in `BulkSlotProvider` interface? or make `BulkSlotProvider` extend `SlotOwner` later.
If `cancelSlotRequest` is basically only used to release slots, then maybe we could rename it later to `releaseSlot` in `BulkSlotProvider`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotWillBeOccupiedIndefinitely);
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProvider.allocatePhysicalSlots(physicalSlotRequests, allocationTimeout))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
Review comment:
Maybe, this could be `SingleLogicalSlot#createAndAllocateFromPhysicalSlot`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
Review comment:
```suggestion
OneSlotPerExecutionSlotAllocator(
```
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -216,13 +221,37 @@ private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
Time timeout) {
slotAllocationRequests.add(Tuple3.of(slotRequestId, task, slotProfile));
- if (slotAllocationDisabled) {
+ if (forceFailingSlotAllocation) {
+ return FutureUtils.completedExceptionally(new Exception("Forced failure"));
+ } else if (slotAllocationDisabled) {
return new CompletableFuture<>();
} else {
return CompletableFuture.completedFuture(new TestingLogicalSlotBuilder().createTestingLogicalSlot());
}
}
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
Review comment:
If we would manage to make `OneSlotPerExecutionSlotAllocator` depending on `BulkSlotProvider`, this `allocatePhysicalSlots` could in a separate `AllocationToggableBulkSlotProvider` or so, as a separate commit if we do it.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -200,14 +203,16 @@ private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
executionVertexId)));
}
- private static class AllocationToggableSlotProvider implements SlotProvider {
+ static class AllocationToggableSlotProvider implements SlotProvider {
private final List<Tuple3<SlotRequestId, ScheduledUnit, SlotProfile>> slotAllocationRequests = new ArrayList<>();
private final List<SlotRequestId> cancelledSlotRequestIds = new ArrayList<>();
private boolean slotAllocationDisabled;
+ private boolean forceFailingSlotAllocation = false;
Review comment:
```suggestion
private boolean forceFailingSlotAllocation;
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocatorFactory.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Factory for {@link OneSlotPerExecutionSlotAllocator}.
+ */
+public class OneSlotPerExecutionSlotAllocatorFactory implements ExecutionSlotAllocatorFactory {
+
+ private final SlotProvider slotProvider;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocatorFactory(
Review comment:
```suggestion
OneSlotPerExecutionSlotAllocatorFactory(
```
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -189,7 +192,7 @@ private DefaultExecutionSlotAllocator createExecutionSlotAllocator(InputsLocatio
return schedulingRequirements;
}
- private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
+ static SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
Review comment:
nit: would be nice to have a separate static util class for the shared static methods
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotWillBeOccupiedIndefinitely);
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProvider.allocatePhysicalSlots(physicalSlotRequests, allocationTimeout))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
Review comment:
```suggestion
executionVertexSlotRequestIds.values()
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
Review comment:
```
private CompletableFuture<SlotProfile> getSlotProfileFuture(
Set<ExecutionVertexID> allExecutionVertexIds,
Set<AllocationID> allPreviousAllocationIds,
ExecutionVertexSchedulingRequirements schedulingRequirements) {
final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
calculatePreferredLocations(
schedulingRequirements.getExecutionVertexId(),
allExecutionVertexIds);
// use the task resource profile as the physical slot resource requirement since slot sharing is ignored
return preferredLocationsFuture.thenApply(
preferredLocations ->
SlotProfile.priorAllocation(
schedulingRequirements.getTaskResourceProfile(),
schedulingRequirements.getTaskResourceProfile(),
preferredLocations,
Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
allPreviousAllocationIds));
}
```
Maybe even deduplicate it with `DefaultExecutionSlotAllocator` by adding an additional `ResourceProfile physicalSlotResourceProfile` arg.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -200,14 +203,16 @@ private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
executionVertexId)));
}
- private static class AllocationToggableSlotProvider implements SlotProvider {
+ static class AllocationToggableSlotProvider implements SlotProvider {
private final List<Tuple3<SlotRequestId, ScheduledUnit, SlotProfile>> slotAllocationRequests = new ArrayList<>();
private final List<SlotRequestId> cancelledSlotRequestIds = new ArrayList<>();
private boolean slotAllocationDisabled;
+ private boolean forceFailingSlotAllocation = false;
Review comment:
nit: it would be nice to introduce `enum AllocationTypeEnum {ENABLED, DISABLED, FAILING}`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
Review comment:
```suggestion
final Map<SlotRequestId, SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements);
final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = slotExecutionVertexAssignments
.entrySet()
.stream()
.collect(Collectors.toMap(e -> e.getValue().getExecutionVertexId(), Map.Entry::getKey));
final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
createPhysicalSlotRequestFutures(
executionVertexSchedulingRequirements,
executionVertexSlotRequestIds);
allocateSlotsForAssignments(
physicalSlotRequestFutures,
slotExecutionVertexAssignments);
return new ArrayList<>(slotExecutionVertexAssignments.values());
```
This seems to be a bit less indexing... It also implies some simplification of the called private methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 7d4734369b4b87997cce4f593cfd706b78b212f6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233)
* 41bc7222db9beee142d3b0e80f51ec8fd80c60a6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 265fa7767926382824b2894b2662d3a93a76b176 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885)
* edab8aef00bcac9a530501fa64165f344d7c74e7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 41bc7222db9beee142d3b0e80f51ec8fd80c60a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320)
* a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438860287
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements);
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = slotExecutionVertexAssignments
+ .stream()
+ .collect(
+ Collectors.toMap(
+ SlotExecutionVertexAssignment::getExecutionVertexId,
+ SlotExecutionVertexAssignment::getSlotRequestId));
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments);
+
+ return Collections.unmodifiableList(slotExecutionVertexAssignments);
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ (slotRequestId, throwable) -> slotProvider.cancelSlotRequest(slotRequestId, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(executionVertexId, allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
+
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotWillBeOccupiedIndefinitely);
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments) {
+
+ final Map<SlotRequestId, SlotExecutionVertexAssignment> requestToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getSlotRequestId, Function.identity()));
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProvider.allocatePhysicalSlots(physicalSlotRequests, allocationTimeout))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final SlotExecutionVertexAssignment assignment = requestToAssignmentMapping.get(slotRequestId);
+
+ checkState(assignment != null);
+
+ final LogicalSlot logicalSlot = SingleLogicalSlot.allocateFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot(),
+ Locality.UNKNOWN,
+ slotOwner,
+ slotWillBeOccupiedIndefinitely);
+ assignment.getLogicalSlotFuture().complete(logicalSlot);
+ }
+ })
+ .exceptionally(ex -> {
+ slotExecutionVertexAssignments.forEach(
+ assignment -> assignment.getLogicalSlotFuture().completeExceptionally(ex));
+ return null;
+ });
+ }
+
+ private class OneSlotPerExecutionSlotAllocatorSlotOwner implements SlotOwner {
+
+ @Override
+ public void returnLogicalSlot(LogicalSlot logicalSlot) {
+ slotProvider.cancelSlotRequest(
+ logicalSlot.getSlotRequestId(),
+ new FlinkException("Slot is being returned to OneSlotPerExecutionSlotAllocator."));
+ }
+ }
Review comment:
I think you are right. done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f009698034f1c67e698a83e43bbb2bcc160f264d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227)
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 7d4734369b4b87997cce4f593cfd706b78b212f6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r431025011
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotOwner slotOwner;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProviderStrategy slotProviderStrategy,
+ final InputsLocationsRetriever inputsLocationsRetriever) {
+ super(slotProviderStrategy, inputsLocationsRetriever);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
Review comment:
Good point. I think it is not supposed to happen with the new thread mode of JM.
I will replace it with a state check as a sanity check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436237778
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ throwable -> slotProvider.cancelSlotRequest(slotRequestId, null, throwable));
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ final CompletableFuture<Collection<TaskManagerLocation>> preferredLocationsFuture =
+ calculatePreferredLocations(
+ executionVertexId,
+ schedulingRequirements.getPreferredLocations(),
+ inputsLocationsRetriever,
+ allExecutionVertexIds);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored
+ final CompletableFuture<SlotProfile> slotProfileFuture = preferredLocationsFuture.thenApply(
+ preferredLocations ->
+ SlotProfile.priorAllocation(
+ schedulingRequirements.getTaskResourceProfile(),
+ schedulingRequirements.getTaskResourceProfile(),
+ preferredLocations,
+ Collections.singletonList(schedulingRequirements.getPreviousAllocationId()),
+ allPreviousAllocationIds));
Review comment:
There is already `ExecutionVertexSchedulingRequirements#physicalSlotResourceProfile` which is the sum of ResourceProfiles of all tasks in the same shared slot.
However, we cannot use it in `OneSlotPerExecutionSlotAllocator` because it ignores slot sharing and the physical slot should fit a single task instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436238944
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -216,13 +221,37 @@ private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
Time timeout) {
slotAllocationRequests.add(Tuple3.of(slotRequestId, task, slotProfile));
- if (slotAllocationDisabled) {
+ if (forceFailingSlotAllocation) {
+ return FutureUtils.completedExceptionally(new Exception("Forced failure"));
+ } else if (slotAllocationDisabled) {
return new CompletableFuture<>();
} else {
return CompletableFuture.completedFuture(new TestingLogicalSlotBuilder().createTestingLogicalSlot());
}
}
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
Review comment:
Yes, `OneSlotPerExecutionSlotAllocatorTest` is reworked and does not use `AllocationToggableBulkSlotProvider` anymore.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -216,13 +221,37 @@ private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
Time timeout) {
slotAllocationRequests.add(Tuple3.of(slotRequestId, task, slotProfile));
- if (slotAllocationDisabled) {
+ if (forceFailingSlotAllocation) {
+ return FutureUtils.completedExceptionally(new Exception("Forced failure"));
+ } else if (slotAllocationDisabled) {
return new CompletableFuture<>();
} else {
return CompletableFuture.completedFuture(new TestingLogicalSlotBuilder().createTestingLogicalSlot());
}
}
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
Review comment:
Yes, `OneSlotPerExecutionSlotAllocatorTest` is reworked now and does not use `AllocationToggableBulkSlotProvider` anymore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429758351
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequestBulk.java
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Represents a bulk of physical slot requests.
+ */
+public class PhysicalSlotRequestBulk {
+
+ final Map<SlotRequestId, PhysicalSlotRequest> pendingRequests;
Review comment:
ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438859134
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocator.java
##########
@@ -36,41 +35,28 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
-import java.util.Objects;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
-import java.util.stream.Collectors;
import static org.apache.flink.util.Preconditions.checkNotNull;
-import static org.apache.flink.util.Preconditions.checkState;
/**
* Default {@link ExecutionSlotAllocator} which will use {@link SlotProvider} to allocate slots and
* keep the unfulfilled requests for further cancellation.
*/
-public class DefaultExecutionSlotAllocator implements ExecutionSlotAllocator {
+public class DefaultExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
private static final Logger LOG = LoggerFactory.getLogger(DefaultExecutionSlotAllocator.class);
- /**
- * Store the uncompleted slot assignments.
- */
- private final Map<ExecutionVertexID, SlotExecutionVertexAssignment> pendingSlotAssignments;
-
private final SlotProviderStrategy slotProviderStrategy;
- private final PreferredLocationsRetriever preferredLocationsRetriever;
-
public DefaultExecutionSlotAllocator(
final SlotProviderStrategy slotProviderStrategy,
final PreferredLocationsRetriever preferredLocationsRetriever) {
- this.slotProviderStrategy = checkNotNull(slotProviderStrategy);
- this.preferredLocationsRetriever = checkNotNull(preferredLocationsRetriever);
- pendingSlotAssignments = new HashMap<>();
+ super(preferredLocationsRetriever);
Review comment:
Opened a separate hotfix to make `DefaultExecutionSlotAllocator` and its factory package private.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-643007617
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f16b12b9c89b723a6c201da252d65e259a53c84 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r429758258
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequest.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+
+/**
+ * Represents a request for a physical slot.
+ */
+public class PhysicalSlotRequest {
+
+ private SlotRequestId slotRequestId;
+
+ private SlotProfile slotProfile;
+
+ private boolean slotWillBeOccupiedIndefinitely;
+
+ public PhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile,
+ final boolean slotWillBeOccupiedIndefinitely) {
+
+ this.slotRequestId = slotRequestId;
+ this.slotProfile = slotProfile;
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ }
+
+ public SlotRequestId getSlotRequestId() {
+ return slotRequestId;
+ }
+
+ public SlotProfile getSlotProfile() {
+ return slotProfile;
+ }
+
+ boolean willSlotBeOccupiedIndefinitely() {
+ return slotWillBeOccupiedIndefinitely;
+ }
+
+ /**
+ * Result of a {@link PhysicalSlotRequest}.
+ */
+ public static class Result {
+
+ private SlotRequestId slotRequestId;
Review comment:
ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f16b12b9c89b723a6c201da252d65e259a53c84 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123)
* 2aa39a469f86ceae75eafb75194f7c7f82e3a6cc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ba2a2eb65072f71c66055c3f86549a061eb1514 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445)
* ca05fc241f6fd0550a39612b4ddef86b63a304a6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7f16b12b9c89b723a6c201da252d65e259a53c84 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123)
* 2aa39a469f86ceae75eafb75194f7c7f82e3a6cc UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-644476328
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436238809
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
Review comment:
It's done to make `OneSlotPerExecutionSlotAllocator` depend on `BulkSlotProvider`.
I think `cancelSlotRequest` may be better because it also cancels pending physical slot request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* 7d4734369b4b87997cce4f593cfd706b78b212f6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233)
* 41bc7222db9beee142d3b0e80f51ec8fd80c60a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2858",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
}, {
"hash" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3180",
"triggerID" : "5607dc27dad87a19fe195670b6e216bc2850cb2e",
"triggerType" : "PUSH"
}, {
"hash" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3184",
"triggerID" : "9ae83a7f8381e681b2f378e5d8d0fd6d48b3e944",
"triggerType" : "PUSH"
}, {
"hash" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3227",
"triggerID" : "f009698034f1c67e698a83e43bbb2bcc160f264d",
"triggerType" : "PUSH"
}, {
"hash" : "f5939316d96975bea8e877770acb52bc17683087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5939316d96975bea8e877770acb52bc17683087",
"triggerType" : "PUSH"
}, {
"hash" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3233",
"triggerID" : "7d4734369b4b87997cce4f593cfd706b78b212f6",
"triggerType" : "PUSH"
}, {
"hash" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3320",
"triggerID" : "41bc7222db9beee142d3b0e80f51ec8fd80c60a6",
"triggerType" : "PUSH"
}, {
"hash" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401",
"triggerID" : "a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f",
"triggerType" : "PUSH"
}, {
"hash" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530",
"triggerID" : "1253ab0db340f4d3aac238a3c2bc8f8f4abb6941",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5939316d96975bea8e877770acb52bc17683087 UNKNOWN
* a751c48402cd2c23dabb4ebf21fc85e3f0d23b9f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3401)
* 1253ab0db340f4d3aac238a3c2bc8f8f4abb6941 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=3530)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-633867170
Thanks for the comments! @azagrebin
Agreed that we can make new allocator and bulk slot allocation independent from existing `SchedulerImp` and `DefaultExecutionSlotAllocator ` for further evolving. Will do it.
> Also, maybe we could split the PR somehow into e.g. two PRs: bulk slot allocation and OneSlotPerExecutionSlotAllocator to be able to merge smaller chunks of code separately
Sure we can do it. The PR is already in smaller pieces and it would be easy to split the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432004595
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
I just realized that `BulkSlotProvider ` is actually making things more complicated because we will need to change how we create it and its factory in `JobMaster` and `DefaultJobManagerRunnerFactory`.
`BulkSlotProvider` needs to know main thread executor and slot pool, which makes it hard to create it in SchedulerNG internally.
Then we will need to keep `BulkSlotProvider` and `Scheduler`/`SlotProvider` at the same time in JobMaster before we can completely drop `SchedulerImpl`.
I think it would be easier to just keep the bulk slot allocation logics `SchedulerImpl`. We can try to make them less coupled with existing logics that will be dropped in the future.
WDYT? @azagrebin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432270401
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.executiongraph.SlotProviderStrategy;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends DefaultExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotOwner slotOwner;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProviderStrategy slotProviderStrategy,
+ final InputsLocationsRetriever inputsLocationsRetriever) {
+ super(slotProviderStrategy, inputsLocationsRetriever);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private static void validateNoCoLocationConstraint(
+ final Collection<ExecutionVertexSchedulingRequirements> schedulingRequirements) {
+
+ final boolean hasCoLocationConstraint = schedulingRequirements.stream()
+ .anyMatch(r -> r.getCoLocationConstraint() != null);
+ checkState(
+ !hasCoLocationConstraint,
+ "Jobs with co-location constraints are not allowed to run with pipelined region scheduling strategy.");
+ }
+
+ private Map<ExecutionVertexID, SlotRequestId> generateExecutionVertexSlotRequestIds(
+ final Set<ExecutionVertexID> allExecutionVertexIds) {
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds = new HashMap<>();
+ for (ExecutionVertexID executionVertexId : allExecutionVertexIds) {
+ executionVertexSlotRequestIds.put(executionVertexId, new SlotRequestId());
+ }
+ return executionVertexSlotRequestIds;
+ }
+
+ private List<SlotExecutionVertexAssignment> createSlotExecutionVertexAssignments(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment =
+ createAndRegisterSlotExecutionVertexAssignment(
+ executionVertexId,
+ new CompletableFuture<>(),
+ slotRequestId,
+ null);
+ slotExecutionVertexAssignments.add(slotExecutionVertexAssignment);
+ }
+
+ return slotExecutionVertexAssignments;
+ }
+
+ private List<CompletableFuture<PhysicalSlotRequest>> createPhysicalSlotRequestFutures(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements,
+ final Set<ExecutionVertexID> allExecutionVertexIds,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Set<AllocationID> allPreviousAllocationIds =
+ computeAllPriorAllocationIds(executionVertexSchedulingRequirements);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ new ArrayList<>(executionVertexSchedulingRequirements.size());
+ for (ExecutionVertexSchedulingRequirements schedulingRequirements : executionVertexSchedulingRequirements) {
+ final ExecutionVertexID executionVertexId = schedulingRequirements.getExecutionVertexId();
+ final SlotRequestId slotRequestId = executionVertexSlotRequestIds.get(executionVertexId);
+
+ LOG.debug("Allocate slot with id {} for execution {}", slotRequestId, executionVertexId);
+
+ // use the task resource profile as the physical slot resource requirement since slot sharing is ignored here
+ final ResourceProfile physicalSlotResourceProfile = schedulingRequirements.getTaskResourceProfile();
+
+ final CompletableFuture<SlotProfile> slotProfileFuture = createSlotProfile(
+ schedulingRequirements,
+ allExecutionVertexIds,
+ physicalSlotResourceProfile,
+ allPreviousAllocationIds);
+ final CompletableFuture<PhysicalSlotRequest> physicalSlotRequestFuture =
+ slotProfileFuture.thenApply(
+ slotProfile -> createPhysicalSlotRequest(slotRequestId, slotProfile));
+ physicalSlotRequestFutures.add(physicalSlotRequestFuture);
+ }
+
+ return physicalSlotRequestFutures;
+ }
+
+ private PhysicalSlotRequest createPhysicalSlotRequest(
+ final SlotRequestId slotRequestId,
+ final SlotProfile slotProfile) {
+ return new PhysicalSlotRequest(slotRequestId, slotProfile, slotProviderStrategy.willSlotBeOccupiedIndefinitely());
+ }
+
+ private void allocateSlotsForAssignments(
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures,
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments,
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds) {
+
+ final Map<ExecutionVertexID, SlotExecutionVertexAssignment> vertexToAssignmentMapping =
+ slotExecutionVertexAssignments.stream()
+ .collect(Collectors.toMap(SlotExecutionVertexAssignment::getExecutionVertexId, Function.identity()));
+
+ final Map<SlotRequestId, ExecutionVertexID> slotToVertexMapping = new HashMap<>();
+ for (Map.Entry<ExecutionVertexID, SlotRequestId> entry : executionVertexSlotRequestIds.entrySet()) {
+ slotToVertexMapping.put(entry.getValue(), entry.getKey());
+ }
+
+ FutureUtils.combineAll(physicalSlotRequestFutures)
+ .thenCompose(physicalSlotRequests -> slotProviderStrategy.allocatePhysicalSlots(physicalSlotRequests))
+ .thenAccept(physicalSlotRequestResults -> {
+ for (PhysicalSlotRequest.Result result : physicalSlotRequestResults) {
+ final SlotRequestId slotRequestId = result.getSlotRequestId();
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ try {
+ final LogicalSlot logicalSlot = allocateSingleLogicalSlotFromPhysicalSlot(
+ slotRequestId,
+ result.getPhysicalSlot());
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().complete(logicalSlot);
+ } catch (Exception ex) {
+ throw new CompletionException(ex);
+ }
+ }
+ })
+ .exceptionally(ex -> {
+ executionVertexSlotRequestIds.values().stream()
+ .forEach(slotRequestId -> {
+ final ExecutionVertexID executionVertexId = slotToVertexMapping.get(slotRequestId);
+ vertexToAssignmentMapping.get(executionVertexId).getLogicalSlotFuture().completeExceptionally(ex);
+ });
+ return null;
+ });
+ }
+
+ private LogicalSlot allocateSingleLogicalSlotFromPhysicalSlot(
+ final SlotRequestId slotRequestId,
+ final PhysicalSlot physicalSlot) throws FlinkException {
+
+ final SingleLogicalSlot singleTaskSlot = new SingleLogicalSlot(
+ slotRequestId,
+ physicalSlot,
+ null,
+ Locality.UNKNOWN,
+ slotOwner);
+
+ if (physicalSlot.tryAssignPayload(singleTaskSlot)) {
+ return singleTaskSlot;
+ } else {
+ throw new FlinkException("Could not assign payload to allocated slot " + physicalSlot.getAllocationId() + '.');
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-643052813
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12256: [FLINK-17018][runtime][WIP] DefaultExecutionSlotAllocator allocates slots in bulks
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 265fa7767926382824b2894b2662d3a93a76b176 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438079017
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
+ final InputsLocationsRetriever inputsLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(inputsLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+
+ this.slotOwner = new OneSlotPerExecutionSlotAllocatorSlotOwner();
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final Set<ExecutionVertexID> allExecutionVertexIds = executionVertexSchedulingRequirements.stream()
+ .map(ExecutionVertexSchedulingRequirements::getExecutionVertexId)
+ .collect(Collectors.toSet());
+
+ final Map<ExecutionVertexID, SlotRequestId> executionVertexSlotRequestIds =
+ generateExecutionVertexSlotRequestIds(allExecutionVertexIds);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ createSlotExecutionVertexAssignments(executionVertexSchedulingRequirements, executionVertexSlotRequestIds);
+
+ final List<CompletableFuture<PhysicalSlotRequest>> physicalSlotRequestFutures =
+ createPhysicalSlotRequestFutures(
+ executionVertexSchedulingRequirements,
+ allExecutionVertexIds,
+ executionVertexSlotRequestIds);
+
+ allocateSlotsForAssignments(
+ physicalSlotRequestFutures,
+ slotExecutionVertexAssignments,
+ executionVertexSlotRequestIds);
+
+ return slotExecutionVertexAssignments;
Review comment:
The API return a list because we want the the given scheduling order to be respected, and thus to avoid disordered scheduling/deployment. See FLINK-14909.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r440243611
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##########
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * <p>1. Slot sharing will be ignored.
+ *
+ * <p>2. Co-location constraints are not allowed.
+ *
+ * <p>3. Intra-bulk input location preferences will be ignored.
+ */
+class OneSlotPerExecutionSlotAllocator extends AbstractExecutionSlotAllocator implements SlotOwner {
+
+ private static final Logger LOG = LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final BulkSlotProvider slotProvider;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ OneSlotPerExecutionSlotAllocator(
+ final BulkSlotProvider slotProvider,
+ final PreferredLocationsRetriever preferredLocationsRetriever,
+ final boolean slotWillBeOccupiedIndefinitely,
+ final Time allocationTimeout) {
+
+ super(preferredLocationsRetriever);
+
+ this.slotProvider = checkNotNull(slotProvider);
+ this.slotWillBeOccupiedIndefinitely = slotWillBeOccupiedIndefinitely;
+ this.allocationTimeout = checkNotNull(allocationTimeout);
+ }
+
+ @Override
+ public List<SlotExecutionVertexAssignment> allocateSlotsFor(
+ final List<ExecutionVertexSchedulingRequirements> executionVertexSchedulingRequirements) {
+
+ validateSchedulingRequirements(executionVertexSchedulingRequirements);
+
+ validateNoCoLocationConstraint(executionVertexSchedulingRequirements);
+
+ final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
Review comment:
The main problem is that the result list order will not match the given list order if we use `new ArrayList<>(slotExecutionVertexAssignments.values())`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432004595
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
I just realized that `BulkSlotProvider ` is actually making things more complicated because we will need to change how we create it and its factory in `JobMaster` and `DefaultJobManagerRunnerFactory`.
`BulkSlotProvider` needs to know main thread executor and slot pool, which makes it hard to create it in SchedulerNG internally.
Then we will need to keep `BulkSlotProvider` and `Scheduler`/`SlotProvider` at the same time in JobMaster before we can completely drop `SchedulerImpl`.
I think it would be easier and cleaner to just keep the bulk slot allocation logics in `SchedulerImpl`. We can try to make them less coupled with existing logics that will be dropped in the future.
WDYT? @azagrebin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438858595
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocatorTest.java
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.instance.SlotSharingGroupId;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraint;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroup;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.BulkSlotProvider;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotTestUtils.createPhysicalSlot;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.createSchedulingRequirements;
+import static org.apache.flink.runtime.scheduler.ExecutionSlotAllocatorTestUtils.findSlotAssignmentByExecutionVertexId;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.junit.Assert.assertThat;
+
+/**
+ * Tests for {@link OneSlotPerExecutionSlotAllocator}.
+ */
+public class OneSlotPerExecutionSlotAllocatorTest extends TestLogger {
+
+ private TestingBulkSlotProvider slotProvider;
+
+ @Before
+ public void setUp() throws Exception {
+ slotProvider = new TestingBulkSlotProvider();
+ }
+
+ @Test
+ public void testSucceededSlotAllocation() {
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(1));
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getExecutionVertexId(), equalTo(executionVertexID));
+ assertThat(slotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(false));
+ }
+
+ @Test
+ public void testFailedSlotAllocation() {
+ final OneSlotPerExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator();
+
+ final ExecutionVertexID executionVertexID = new ExecutionVertexID(new JobVertexID(), 0);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(executionVertexID);
+
+ slotProvider.forceFailingSlotAllocation();
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ final SlotExecutionVertexAssignment slotAssignment =
+ findSlotAssignmentByExecutionVertexId(executionVertexID, slotExecutionVertexAssignments);
+
+ assertThat(slotAssignment.getLogicalSlotFuture().isCompletedExceptionally(), is(true));
+ assertThat(executionSlotAllocator.getPendingSlotAssignments().keySet(), hasSize(0));
+ assertThat(slotProvider.getCancelledSlotRequestIds(), contains(slotAssignment.getSlotRequestId()));
+ }
+
+ @Test
+ public void testInterBulkInputLocationPreferencesAreRespected() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForProducer =
+ createSchedulingRequirements(producerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForProducer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForProducer);
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignmentsForProducer);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+
+ inputsLocationsRetriever.markScheduled(consumerId);
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirementsForConsumer =
+ createSchedulingRequirements(consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignmentsForConsumer =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirementsForConsumer);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignmentsForConsumer);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(false));
+
+ inputsLocationsRetriever.assignTaskManagerLocation(producerId);
+
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testIntraBulkInputLocationPreferencesDoNotBlockAllocation() {
+ final ExecutionVertexID producerId = new ExecutionVertexID(new JobVertexID(), 0);
+ final ExecutionVertexID consumerId = new ExecutionVertexID(new JobVertexID(), 0);
+
+ final TestingInputsLocationsRetriever inputsLocationsRetriever = new TestingInputsLocationsRetriever.Builder()
+ .connectConsumerToProducer(consumerId, producerId)
+ .build();
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ new TestingStateLocationRetriever(),
+ inputsLocationsRetriever);
+
+ inputsLocationsRetriever.markScheduled(producerId);
+ inputsLocationsRetriever.markScheduled(consumerId);
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements =
+ createSchedulingRequirements(producerId, consumerId);
+ final Collection<SlotExecutionVertexAssignment> slotExecutionVertexAssignments =
+ executionSlotAllocator.allocateSlotsFor(schedulingRequirements);
+
+ assertThat(slotExecutionVertexAssignments, hasSize(2));
+
+ final SlotExecutionVertexAssignment producerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(producerId, slotExecutionVertexAssignments);
+ final SlotExecutionVertexAssignment consumerSlotAssignment =
+ findSlotAssignmentByExecutionVertexId(consumerId, slotExecutionVertexAssignments);
+
+ assertThat(producerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ assertThat(consumerSlotAssignment.getLogicalSlotFuture().isDone(), is(true));
+ }
+
+ @Test
+ public void testCreatedSlotRequests() {
+ final ExecutionVertexID executionVertexId = new ExecutionVertexID(new JobVertexID(), 0);
+ final AllocationID allocationId = new AllocationID();
+ final SlotSharingGroupId sharingGroupId = new SlotSharingGroupId();
+ final ResourceProfile taskResourceProfile = ResourceProfile.fromResources(0.5, 250);
+ final ResourceProfile physicalSlotResourceProfile = ResourceProfile.fromResources(1.0, 300);
+ final TaskManagerLocation taskManagerLocation = new LocalTaskManagerLocation();
+
+ final TestingStateLocationRetriever stateLocationRetriever = new TestingStateLocationRetriever();
+ stateLocationRetriever.setStateLocation(executionVertexId, taskManagerLocation);
+
+ final ExecutionSlotAllocator executionSlotAllocator = createExecutionSlotAllocator(
+ stateLocationRetriever,
+ new TestingInputsLocationsRetriever.Builder().build());
+
+ final List<ExecutionVertexSchedulingRequirements> schedulingRequirements = Arrays.asList(
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r436239133
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultExecutionSlotAllocatorTest.java
##########
@@ -200,14 +203,16 @@ private SlotExecutionVertexAssignment findSlotAssignmentByExecutionVertexId(
executionVertexId)));
}
- private static class AllocationToggableSlotProvider implements SlotProvider {
+ static class AllocationToggableSlotProvider implements SlotProvider {
private final List<Tuple3<SlotRequestId, ScheduledUnit, SlotProfile>> slotAllocationRequests = new ArrayList<>();
private final List<SlotRequestId> cancelledSlotRequestIds = new ArrayList<>();
private boolean slotAllocationDisabled;
+ private boolean forceFailingSlotAllocation = false;
Review comment:
`OneSlotPerExecutionSlotAllocatorTest` now has its own `TestingBulkSlotProvider` which does not have 3 modes.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocatorFactory.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Factory for {@link OneSlotPerExecutionSlotAllocator}.
+ */
+public class OneSlotPerExecutionSlotAllocatorFactory implements ExecutionSlotAllocatorFactory {
+
+ private final SlotProvider slotProvider;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocatorFactory(
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r438860474
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/AbstractExecutionSlotAllocator.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BiConsumer;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Base class for all {@link ExecutionSlotAllocator}. It is responsible to allocate slots for tasks and
+ * keep the unfulfilled slot requests for further cancellation.
+ */
+abstract class AbstractExecutionSlotAllocator implements ExecutionSlotAllocator {
+
+ private final Map<ExecutionVertexID, SlotExecutionVertexAssignment> pendingSlotAssignments;
+
+ private final PreferredLocationsRetriever preferredLocationsRetriever;
+
+ protected AbstractExecutionSlotAllocator(final PreferredLocationsRetriever preferredLocationsRetriever) {
+ this.preferredLocationsRetriever = checkNotNull(preferredLocationsRetriever);
+ this.pendingSlotAssignments = new HashMap<>();
+ }
+
+ @Override
+ public void cancel(final ExecutionVertexID executionVertexId) {
+ final SlotExecutionVertexAssignment slotExecutionVertexAssignment = pendingSlotAssignments.get(executionVertexId);
+ if (slotExecutionVertexAssignment != null) {
+ slotExecutionVertexAssignment.getLogicalSlotFuture().cancel(false);
+ }
+ }
+
+ protected void validateSchedulingRequirements(
Review comment:
done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8ba2a2eb65072f71c66055c3f86549a061eb1514 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445)
* ca05fc241f6fd0550a39612b4ddef86b63a304a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r432239813
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##########
@@ -562,4 +575,110 @@ private void releaseSharedSlot(
public boolean requiresPreviousExecutionGraphAllocations() {
return slotSelectionStrategy instanceof PreviousAllocationSlotSelectionStrategy;
}
+
+ @Override
+ public CompletableFuture<Collection<PhysicalSlotRequest.Result>> allocatePhysicalSlots(
+ final Collection<PhysicalSlotRequest> physicalSlotRequests,
+ final Time timeout) {
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List<CompletableFuture<PhysicalSlotRequest.Result>> resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture<PhysicalSlotRequest.Result> resultFuture =
+ allocatePhysicalSlot(request, timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+ result.getPhysicalSlot().getAllocationId());
+
+ return result;
+ });
+ resultFutures.add(resultFuture);
+ }
+
+ slotRequestBulkTracker.track(slotRequestBulk);
+ schedulePendingRequestBulkTimeoutCheck(slotRequestBulk, timeout);
+
+ return FutureUtils.combineAll(resultFutures)
+ .whenComplete((ignore, throwable) -> slotRequestBulkTracker.untrack(slotRequestBulk));
+ }
+
+ private CompletableFuture<PhysicalSlotRequest.Result> allocatePhysicalSlot(
+ final PhysicalSlotRequest physicalSlotRequest,
+ final Time timeout) {
+
+ final SlotRequestId slotRequestId = physicalSlotRequest.getSlotRequestId();
+ final SlotProfile slotProfile = physicalSlotRequest.getSlotProfile();
+
+ final Optional<SlotAndLocality> availablePhysicalSlot = tryAllocateFromAvailable(slotRequestId, slotProfile);
Review comment:
I find one way that we can still have a separate `BulkSlotProvider` and do not need to change `Scheduler`/`SlotProvider ` creating and initializing codes.
That is, we still enables `SlotProvider` for bulk slot allocation. However, `SchedulerImpl` creates a `BulkSlotProviderImpl` internally and dedicates bulk slot allocation requests to it.
See 395c6cba8c463cffe2cf3cc70d91e6c6ff0dbb9f
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #12256:
URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "265fa7767926382824b2894b2662d3a93a76b176",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885",
"triggerID" : "265fa7767926382824b2894b2662d3a93a76b176",
"triggerType" : "PUSH"
}, {
"hash" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916",
"triggerID" : "edab8aef00bcac9a530501fa64165f344d7c74e7",
"triggerType" : "PUSH"
}, {
"hash" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2117",
"triggerID" : "cce8477d3eae8bad8b4109fa1f85e810220b6d39",
"triggerType" : "PUSH"
}, {
"hash" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2123",
"triggerID" : "7f16b12b9c89b723a6c201da252d65e259a53c84",
"triggerType" : "PUSH"
}, {
"hash" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2422",
"triggerID" : "2aa39a469f86ceae75eafb75194f7c7f82e3a6cc",
"triggerType" : "PUSH"
}, {
"hash" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2445",
"triggerID" : "8ba2a2eb65072f71c66055c3f86549a061eb1514",
"triggerType" : "PUSH"
}, {
"hash" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625",
"triggerID" : "ca05fc241f6fd0550a39612b4ddef86b63a304a6",
"triggerType" : "PUSH"
}, {
"hash" : "f181cf943442db6173eba4c817f40bbab49538da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857",
"triggerID" : "f181cf943442db6173eba4c817f40bbab49538da",
"triggerType" : "PUSH"
}, {
"hash" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eba9c31264c2a0fe9f5bf955ba21de081bd1aa25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca05fc241f6fd0550a39612b4ddef86b63a304a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2625)
* f181cf943442db6173eba4c817f40bbab49538da Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2857)
* eba9c31264c2a0fe9f5bf955ba21de081bd1aa25 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org