You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/08 11:03:10 UTC
[GitHub] [flink] zentol opened a new pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
zentol opened a new pull request #13561:
URL: https://github.com/apache/flink/pull/13561
Adds the `AllocatedSlotPool`, a component of the new declarative slot pool, providing basic book-keeping for allocated slots.
With this PR I'm also proposing a change in terminology in regards to `allocate`. Currently, "allocating a slot" has different meanings as it both describes a slot being assigned to a job by the ResourceManager, and an (allocated) slot being allocated for a task by the JobMaster.
I'd like to get rid of this duality, and propose for the job master to "reserve" allocated slots when using them for a task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705507172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292",
"triggerID" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5a1938350a935a8d24ee93c1f83114e65819bc1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705507172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292",
"triggerID" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"triggerType" : "PUSH"
}, {
"hash" : "ce728545af6c30e615d4c972b902d065b454ae18",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7506",
"triggerID" : "ce728545af6c30e615d4c972b902d065b454ae18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5a1938350a935a8d24ee93c1f83114e65819bc1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292)
* ce728545af6c30e615d4c972b902d065b454ae18 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7506)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705497007
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit b5a1938350a935a8d24ee93c1f83114e65819bc1 (Thu Oct 08 11:05:12 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705507172
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13561:
URL: https://github.com/apache/flink/pull/13561#discussion_r503289846
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPool.java
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Default {@link AllocatedSlotPool} implementation.
+ */
+public class DefaultAllocatedSlotPool implements AllocatedSlotPool {
+
+ private final Map<AllocationID, AllocatedSlot> registeredSlots;
+
+ /**
+ * Map containing all free slots and since when they are free.
+ */
+ private final Map<AllocationID, Long> freeSlotsSince;
+
+ /**
+ * Index containing a mapping between TaskExecutors and their slots.
+ */
+ private final Map<ResourceID, Set<AllocationID>> slotsPerTaskExecutor;
+
+ public DefaultAllocatedSlotPool() {
+ this.registeredSlots = new HashMap<>();
+ this.slotsPerTaskExecutor = new HashMap<>();
+ this.freeSlotsSince = new HashMap<>();
+ }
+
+ @Override
+ public void addSlots(Collection<AllocatedSlot> slots, long currentTime) {
+ for (AllocatedSlot slot : slots) {
+ addSlot(slot, currentTime);
+ }
+ }
+
+ private void addSlot(AllocatedSlot slot, long currentTime) {
+ Preconditions.checkState(!registeredSlots.containsKey(slot.getAllocationId()), "The slot pool already contains a slot with id %s", slot.getAllocationId());
+ addSlotInternal(slot, currentTime);
+
+ slotsPerTaskExecutor
+ .computeIfAbsent(slot.getTaskManagerId(), resourceID -> new HashSet<>())
+ .add(slot.getAllocationId());
Review comment:
I think we can remove this part.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPoolTest.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGatewayBuilder;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+import org.junit.Test;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.net.InetAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.Matchers.closeTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.sameInstance;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests for the {@link DefaultAllocatedSlotPool}.
+ */
+public class DefaultAllocatedSlotPoolTest extends TestLogger {
+
+ @Test
+ public void testAddSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ slotPool.addSlots(slots, 0);
+
+ assertSlotPoolContainsSlots(slotPool, slots);
+ assertSlotPoolContainsFreeSlots(slotPool, slots);
+ }
+
+ @Test
+ public void testRemoveSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ slotPool.addSlots(slots, 0);
+
+ final Iterator<AllocatedSlot> iterator = slots.iterator();
+ final AllocatedSlot removedSlot = iterator.next();
+ iterator.remove();
+
+ slotPool.removeSlot(removedSlot.getAllocationId());
+
+ assertSlotPoolContainsSlots(slotPool, slots);
+ }
+
+ @Test
+ public void testRemoveSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final ResourceID owner = ResourceID.generate();
+ final Collection<AllocatedSlot> slots = createAllocatedSlotsWithOwner(owner);
+ final AllocatedSlot otherSlot = createAllocatedSlot(ResourceID.generate());
+ slots.add(otherSlot);
+
+ slotPool.addSlots(slots, 0);
+
+ slotPool.removeSlots(owner);
+
+ assertSlotPoolContainsSlots(slotPool, Collections.singleton(otherSlot));
+ }
+
+ @Test
+ public void testContainsSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final ResourceID owner = ResourceID.generate();
+ final AllocatedSlot allocatedSlot = createAllocatedSlot(owner);
+
+ slotPool.addSlots(Collections.singleton(allocatedSlot), 0);
+
+ assertTrue(slotPool.containsSlots(owner));
+ assertFalse(slotPool.containsSlots(ResourceID.generate()));
+ }
+
+ @Test
+ public void testContainsSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final AllocatedSlot allocatedSlot = createAllocatedSlot(null);
+
+ slotPool.addSlots(Collections.singleton(allocatedSlot), 0);
+
+ assertTrue(slotPool.containsSlot(allocatedSlot.getAllocationId()));
+ assertFalse(slotPool.containsSlot(new AllocationID()));
+ }
+
+ @Test
+ public void testReserveFreeSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final Collection<AllocatedSlot> allSlots = createAllocatedSlots();
+ final Collection<AllocatedSlot> freeSlots = new ArrayList<>(allSlots);
+ final Iterator<AllocatedSlot> iterator = freeSlots.iterator();
+ final AllocatedSlot allocatedSlot = iterator.next();
+ iterator.remove();
+
+ slotPool.addSlots(allSlots, 0);
+
+ assertThat(slotPool.reserveFreeSlot(allocatedSlot.getAllocationId()), sameInstance(allocatedSlot));
+
+ assertSlotPoolContainsFreeSlots(slotPool, freeSlots);
+ assertSlotPoolContainsSlots(slotPool, allSlots);
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testReserveNonFreeSlotFails() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final AllocatedSlot slot = createAllocatedSlot(null);
+
+ slotPool.addSlots(Collections.singleton(slot), 0);
+
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+ }
+
+ @Test
+ public void testFreeReservedSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ final int initialTime = 0;
+ slotPool.addSlots(slots, initialTime);
+
+ final AllocatedSlot slot = slots.iterator().next();
+
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+
+ final int releaseTime = 1;
+ assertTrue(slotPool.freeReservedSlot(slot.getAllocationId(), releaseTime).isPresent());
+ assertSlotPoolContainsFreeSlots(slotPool, slots);
+
+ for (AllocatedSlotPool.FreeSlotInfo freeSlotInfo : slotPool.getFreeSlotsInformation()) {
+ final long time;
+ if (freeSlotInfo.getAllocationId().equals(slot.getAllocationId())) {
+ time = releaseTime;
+ } else {
+ time = initialTime;
+ }
+
+ assertThat(freeSlotInfo.getFreeSince(), is(time));
+ }
+ }
+
+ @Test
+ public void testFreeFreeSlotIsIgnored() {
Review comment:
```suggestion
public void testFreeReservedSlotIsIgnored() {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13561:
URL: https://github.com/apache/flink/pull/13561#discussion_r503767564
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPoolTest.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGatewayBuilder;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+import org.junit.Test;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.net.InetAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.Matchers.closeTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.sameInstance;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests for the {@link DefaultAllocatedSlotPool}.
+ */
+public class DefaultAllocatedSlotPoolTest extends TestLogger {
+
+ @Test
+ public void testAddSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ slotPool.addSlots(slots, 0);
+
+ assertSlotPoolContainsSlots(slotPool, slots);
+ assertSlotPoolContainsFreeSlots(slotPool, slots);
+ }
+
+ @Test
+ public void testRemoveSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ slotPool.addSlots(slots, 0);
+
+ final Iterator<AllocatedSlot> iterator = slots.iterator();
+ final AllocatedSlot removedSlot = iterator.next();
+ iterator.remove();
+
+ slotPool.removeSlot(removedSlot.getAllocationId());
+
+ assertSlotPoolContainsSlots(slotPool, slots);
+ }
+
+ @Test
+ public void testRemoveSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final ResourceID owner = ResourceID.generate();
+ final Collection<AllocatedSlot> slots = createAllocatedSlotsWithOwner(owner);
+ final AllocatedSlot otherSlot = createAllocatedSlot(ResourceID.generate());
+ slots.add(otherSlot);
+
+ slotPool.addSlots(slots, 0);
+
+ slotPool.removeSlots(owner);
+
+ assertSlotPoolContainsSlots(slotPool, Collections.singleton(otherSlot));
+ }
+
+ @Test
+ public void testContainsSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final ResourceID owner = ResourceID.generate();
+ final AllocatedSlot allocatedSlot = createAllocatedSlot(owner);
+
+ slotPool.addSlots(Collections.singleton(allocatedSlot), 0);
+
+ assertTrue(slotPool.containsSlots(owner));
+ assertFalse(slotPool.containsSlots(ResourceID.generate()));
+ }
+
+ @Test
+ public void testContainsSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final AllocatedSlot allocatedSlot = createAllocatedSlot(null);
+
+ slotPool.addSlots(Collections.singleton(allocatedSlot), 0);
+
+ assertTrue(slotPool.containsSlot(allocatedSlot.getAllocationId()));
+ assertFalse(slotPool.containsSlot(new AllocationID()));
+ }
+
+ @Test
+ public void testReserveFreeSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final Collection<AllocatedSlot> allSlots = createAllocatedSlots();
+ final Collection<AllocatedSlot> freeSlots = new ArrayList<>(allSlots);
+ final Iterator<AllocatedSlot> iterator = freeSlots.iterator();
+ final AllocatedSlot allocatedSlot = iterator.next();
+ iterator.remove();
+
+ slotPool.addSlots(allSlots, 0);
+
+ assertThat(slotPool.reserveFreeSlot(allocatedSlot.getAllocationId()), sameInstance(allocatedSlot));
+
+ assertSlotPoolContainsFreeSlots(slotPool, freeSlots);
+ assertSlotPoolContainsSlots(slotPool, allSlots);
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testReserveNonFreeSlotFails() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final AllocatedSlot slot = createAllocatedSlot(null);
+
+ slotPool.addSlots(Collections.singleton(slot), 0);
+
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+ }
+
+ @Test
+ public void testFreeReservedSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ final int initialTime = 0;
+ slotPool.addSlots(slots, initialTime);
+
+ final AllocatedSlot slot = slots.iterator().next();
+
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+
+ final int releaseTime = 1;
+ assertTrue(slotPool.freeReservedSlot(slot.getAllocationId(), releaseTime).isPresent());
+ assertSlotPoolContainsFreeSlots(slotPool, slots);
+
+ for (AllocatedSlotPool.FreeSlotInfo freeSlotInfo : slotPool.getFreeSlotsInformation()) {
+ final long time;
+ if (freeSlotInfo.getAllocationId().equals(slot.getAllocationId())) {
+ time = releaseTime;
+ } else {
+ time = initialTime;
+ }
+
+ assertThat(freeSlotInfo.getFreeSince(), is(time));
+ }
+ }
+
+ @Test
+ public void testFreeFreeSlotIsIgnored() {
Review comment:
This test checks that freeing a free slot does not cause any side-effects, so the proposed name would be wrong. The method name is indeed a bit awkward; an alternative might be `testFreeingFreeSlotIsIgnored`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705497007
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol merged pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
zentol merged pull request #13561:
URL: https://github.com/apache/flink/pull/13561
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705507172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292",
"triggerID" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"triggerType" : "PUSH"
}, {
"hash" : "ce728545af6c30e615d4c972b902d065b454ae18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce728545af6c30e615d4c972b902d065b454ae18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5a1938350a935a8d24ee93c1f83114e65819bc1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292)
* ce728545af6c30e615d4c972b902d065b454ae18 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #13561:
URL: https://github.com/apache/flink/pull/13561#discussion_r503770505
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPoolTest.java
##########
@@ -0,0 +1,315 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGatewayBuilder;
+import org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.TestLogger;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+import org.junit.Test;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.net.InetAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.Matchers.closeTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.sameInstance;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Tests for the {@link DefaultAllocatedSlotPool}.
+ */
+public class DefaultAllocatedSlotPoolTest extends TestLogger {
+
+ @Test
+ public void testAddSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ slotPool.addSlots(slots, 0);
+
+ assertSlotPoolContainsSlots(slotPool, slots);
+ assertSlotPoolContainsFreeSlots(slotPool, slots);
+ }
+
+ @Test
+ public void testRemoveSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ slotPool.addSlots(slots, 0);
+
+ final Iterator<AllocatedSlot> iterator = slots.iterator();
+ final AllocatedSlot removedSlot = iterator.next();
+ iterator.remove();
+
+ slotPool.removeSlot(removedSlot.getAllocationId());
+
+ assertSlotPoolContainsSlots(slotPool, slots);
+ }
+
+ @Test
+ public void testRemoveSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+
+ final ResourceID owner = ResourceID.generate();
+ final Collection<AllocatedSlot> slots = createAllocatedSlotsWithOwner(owner);
+ final AllocatedSlot otherSlot = createAllocatedSlot(ResourceID.generate());
+ slots.add(otherSlot);
+
+ slotPool.addSlots(slots, 0);
+
+ slotPool.removeSlots(owner);
+
+ assertSlotPoolContainsSlots(slotPool, Collections.singleton(otherSlot));
+ }
+
+ @Test
+ public void testContainsSlots() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final ResourceID owner = ResourceID.generate();
+ final AllocatedSlot allocatedSlot = createAllocatedSlot(owner);
+
+ slotPool.addSlots(Collections.singleton(allocatedSlot), 0);
+
+ assertTrue(slotPool.containsSlots(owner));
+ assertFalse(slotPool.containsSlots(ResourceID.generate()));
+ }
+
+ @Test
+ public void testContainsSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final AllocatedSlot allocatedSlot = createAllocatedSlot(null);
+
+ slotPool.addSlots(Collections.singleton(allocatedSlot), 0);
+
+ assertTrue(slotPool.containsSlot(allocatedSlot.getAllocationId()));
+ assertFalse(slotPool.containsSlot(new AllocationID()));
+ }
+
+ @Test
+ public void testReserveFreeSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final Collection<AllocatedSlot> allSlots = createAllocatedSlots();
+ final Collection<AllocatedSlot> freeSlots = new ArrayList<>(allSlots);
+ final Iterator<AllocatedSlot> iterator = freeSlots.iterator();
+ final AllocatedSlot allocatedSlot = iterator.next();
+ iterator.remove();
+
+ slotPool.addSlots(allSlots, 0);
+
+ assertThat(slotPool.reserveFreeSlot(allocatedSlot.getAllocationId()), sameInstance(allocatedSlot));
+
+ assertSlotPoolContainsFreeSlots(slotPool, freeSlots);
+ assertSlotPoolContainsSlots(slotPool, allSlots);
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testReserveNonFreeSlotFails() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final AllocatedSlot slot = createAllocatedSlot(null);
+
+ slotPool.addSlots(Collections.singleton(slot), 0);
+
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+ }
+
+ @Test
+ public void testFreeReservedSlot() {
+ final DefaultAllocatedSlotPool slotPool = new DefaultAllocatedSlotPool();
+ final Collection<AllocatedSlot> slots = createAllocatedSlots();
+
+ final int initialTime = 0;
+ slotPool.addSlots(slots, initialTime);
+
+ final AllocatedSlot slot = slots.iterator().next();
+
+ slotPool.reserveFreeSlot(slot.getAllocationId());
+
+ final int releaseTime = 1;
+ assertTrue(slotPool.freeReservedSlot(slot.getAllocationId(), releaseTime).isPresent());
+ assertSlotPoolContainsFreeSlots(slotPool, slots);
+
+ for (AllocatedSlotPool.FreeSlotInfo freeSlotInfo : slotPool.getFreeSlotsInformation()) {
+ final long time;
+ if (freeSlotInfo.getAllocationId().equals(slot.getAllocationId())) {
+ time = releaseTime;
+ } else {
+ time = initialTime;
+ }
+
+ assertThat(freeSlotInfo.getFreeSince(), is(time));
+ }
+ }
+
+ @Test
+ public void testFreeFreeSlotIsIgnored() {
Review comment:
Ah ok, then let's call `testFreeingFreeSlotIsIgnored`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705507172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5a1938350a935a8d24ee93c1f83114e65819bc1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #13561:
URL: https://github.com/apache/flink/pull/13561#discussion_r503769706
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPool.java
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.util.Preconditions;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Default {@link AllocatedSlotPool} implementation.
+ */
+public class DefaultAllocatedSlotPool implements AllocatedSlotPool {
+
+ private final Map<AllocationID, AllocatedSlot> registeredSlots;
+
+ /**
+ * Map containing all free slots and since when they are free.
+ */
+ private final Map<AllocationID, Long> freeSlotsSince;
+
+ /**
+ * Index containing a mapping between TaskExecutors and their slots.
+ */
+ private final Map<ResourceID, Set<AllocationID>> slotsPerTaskExecutor;
+
+ public DefaultAllocatedSlotPool() {
+ this.registeredSlots = new HashMap<>();
+ this.slotsPerTaskExecutor = new HashMap<>();
+ this.freeSlotsSince = new HashMap<>();
+ }
+
+ @Override
+ public void addSlots(Collection<AllocatedSlot> slots, long currentTime) {
+ for (AllocatedSlot slot : slots) {
+ addSlot(slot, currentTime);
+ }
+ }
+
+ private void addSlot(AllocatedSlot slot, long currentTime) {
+ Preconditions.checkState(!registeredSlots.containsKey(slot.getAllocationId()), "The slot pool already contains a slot with id %s", slot.getAllocationId());
+ addSlotInternal(slot, currentTime);
+
+ slotsPerTaskExecutor
+ .computeIfAbsent(slot.getTaskManagerId(), resourceID -> new HashSet<>())
+ .add(slot.getAllocationId());
Review comment:
argh, I will remove the copy in `addSlotInternal`. Leftover from an experiment where all 3 data-structure updates were done in `addSlotInternal`, but then you'd lose symmetry with `removeSlot(Internal)`, which you could only avoid by moving _most_ of `removeSlot`, but then there was little benefit...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705507172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292",
"triggerID" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5a1938350a935a8d24ee93c1f83114e65819bc1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13561: [FLINK-19315][coordination] Add AllocatedSlotPool
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13561:
URL: https://github.com/apache/flink/pull/13561#issuecomment-705507172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7292",
"triggerID" : "b5a1938350a935a8d24ee93c1f83114e65819bc1",
"triggerType" : "PUSH"
}, {
"hash" : "ce728545af6c30e615d4c972b902d065b454ae18",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7506",
"triggerID" : "ce728545af6c30e615d4c972b902d065b454ae18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ce728545af6c30e615d4c972b902d065b454ae18 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7506)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org