You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2017/11/27 17:20:13 UTC

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/5088

    [FLINK-8087] Decouple Slot from AllocatedSlot

    ## What is the purpose of the change
    
    This commit introduces the SlotContext which is an abstraction for the SimpleSlot
    to obtain the relevant slot information to do the communication with the
    TaskManager without relying on the AllocatedSlot which is now only used by the
    SlotPool.
    
    This PR is based on #5087. 
    
    ## Brief change log
    
    - Introduce `SlotContext` as simple abstraction for slot related information
    - Remove dependency of `Slot` on `AllocatedSlot` which is now only used internally by the `SlotPool`.
    - Introduce `SimpleSlotContext` which implements `SlotContext` and acts as the slot context for the `SimpleSlot`.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)
    
    CC: @GJL 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink decoupleSlotFromSlotPool

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5088.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5088
    
----
commit d30dde83548dbeff4249f3b57b67cdb6247af510
Author: Till Rohrmann <tr...@apache.org>
Date:   2017-11-14T22:50:52Z

    [FLINK-8078] Introduce LogicalSlot interface
    
    The LogicalSlot interface decouples the task deployment from the actual
    slot implementation which at the moment is Slot, SimpleSlot and SharedSlot.
    This is a helpful step to introduce a different slot implementation for
    Flip-6.

commit e5da9566a6fc8a36ac8b06bae911c0dff5554e5d
Author: Till Rohrmann <tr...@apache.org>
Date:   2017-11-15T13:20:27Z

    [FLINK-8085] Thin out LogicalSlot interface
    
    Remove isCanceled, isReleased method and decouple logical slot from Execution by
    introducing a Payload interface which is set for a LogicalSlot. The Payload interface
    is implemented by the Execution and allows to fail an implementation and obtaining
    a termination future.
    
    Introduce proper Execution#releaseFuture which is completed once the Execution's
    assigned resource has been released.

commit 84d86bebe2f9f8395430e7c71dd2393ba117b44f
Author: Till Rohrmann <tr...@apache.org>
Date:   2017-11-24T17:03:49Z

    [FLINK-8087] Decouple Slot from AllocatedSlot
    
    This commit introduces the SlotContext which is an abstraction for the SimpleSlot
    to obtain the relevant slot information to do the communication with the
    TaskManager without relying on the AllocatedSlot which is now only used by the
    SlotPool.

----


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r155279729
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/instance/SlotPool.java ---
    @@ -312,24 +312,36 @@ public void returnAllocatedSlot(Slot slot) {
     		// (1) do we have a slot available already?
     		SlotAndLocality slotFromPool = availableSlots.poll(resources, locationPreferences);
     		if (slotFromPool != null) {
    -			SimpleSlot slot = createSimpleSlot(slotFromPool.slot(), slotFromPool.locality());
    --- End diff --
    
    Method `createSimpleSlot` is no longer in use.


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r155258087
  
    --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/instance/SlotsTestImplContext.java ---
    @@ -31,7 +30,7 @@
     import static org.mockito.Mockito.mock;
     import static org.mockito.Mockito.when;
     
    -public class AllocatedSlotsTest {
    +public class SlotsTestImplContext {
    --- End diff --
    
    Why was this renamed?


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5088


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r156880286
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/instance/SlotPool.java ---
    @@ -312,24 +312,36 @@ public void returnAllocatedSlot(Slot slot) {
     		// (1) do we have a slot available already?
     		SlotAndLocality slotFromPool = availableSlots.poll(resources, locationPreferences);
     		if (slotFromPool != null) {
    -			SimpleSlot slot = createSimpleSlot(slotFromPool.slot(), slotFromPool.locality());
    --- End diff --
    
    True. Will do it.


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r155296898
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/instance/AllocatedSlot.java ---
    @@ -144,6 +144,78 @@ public TaskManagerGateway getTaskManagerGateway() {
     		return taskManagerGateway;
     	}
     
    +	/**
    +	 * Triggers the release of the logical slot.
    +	 */
    +	public void triggerLogicalSlotRelease() {
    +		final LogicalSlot logicalSlot = logicalSlotReference.get();
    +
    +		if (logicalSlot != null) {
    +			logicalSlot.releaseSlot();
    +		}
    +	}
    +
    +	/**
    +	 * Releases the logical slot.
    +	 *
    +	 * @return true if the logical slot could be released, false otherwise.
    +	 */
    +	public boolean releaseLogicalSlot() {
    +		final LogicalSlot logicalSlot = logicalSlotReference.get();
    +
    +		if (logicalSlot != null) {
    +			if (logicalSlot instanceof Slot) {
    +				final Slot slot = (Slot) logicalSlot;
    +				if (slot.markReleased()) {
    +					logicalSlotReference.set(null);
    +					return true;
    +				}
    +			} else {
    +				throw new RuntimeException("Unsupported logical slot type encounterd " + logicalSlot.getClass());
    --- End diff --
    
    Typo: * encounterd* 


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r156880229
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/slots/SlotContext.java ---
    @@ -0,0 +1,61 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.runtime.jobmanager.slots;
    +
    +import org.apache.flink.runtime.clusterframework.types.AllocationID;
    +import org.apache.flink.runtime.instance.Slot;
    +import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
    +
    +/**
    + * Interface for the context of a logical {@link Slot}. This context contains information
    + * about the underlying allocated slot and how to communicate with the TaskManager on which
    + * it was allocated.
    + */
    +public interface SlotContext {
    +
    +	/**
    +	 * Gets the ID under which the slot is allocated, which uniquely identifies the slot.
    +	 *
    +	 * @return The ID under which the slot is allocated
    +	 */
    +	AllocationID getAllocationId();
    +
    +	/**
    +	 * Gets the location info of the TaskManager that offers this slot.
    +	 *
    +	 * @return The location info of the TaskManager that offers this slot
    +	 */
    +	TaskManagerLocation getTaskManagerLocation();
    +
    +	/**
    +	 * Gets the number of the slot.
    +	 *
    +	 * @return The number of the slot on the TaskManager.
    +	 */
    +	int getPhysicalSlotNumber();
    +
    +	/**
    +	 * Gets the actor gateway that can be used to send messages to the TaskManager.
    +	 * <p>
    +	 * This method should be removed once the new interface-based RPC abstraction is in place
    +	 *
    +	 * @return The actor gateway that can be used to send messages to the TaskManager.
    --- End diff --
    
    Will remove it.


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by GJL <gi...@git.apache.org>.
Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r155276432
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/slots/SlotContext.java ---
    @@ -0,0 +1,61 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.runtime.jobmanager.slots;
    +
    +import org.apache.flink.runtime.clusterframework.types.AllocationID;
    +import org.apache.flink.runtime.instance.Slot;
    +import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
    +
    +/**
    + * Interface for the context of a logical {@link Slot}. This context contains information
    + * about the underlying allocated slot and how to communicate with the TaskManager on which
    + * it was allocated.
    + */
    +public interface SlotContext {
    +
    +	/**
    +	 * Gets the ID under which the slot is allocated, which uniquely identifies the slot.
    +	 *
    +	 * @return The ID under which the slot is allocated
    +	 */
    +	AllocationID getAllocationId();
    +
    +	/**
    +	 * Gets the location info of the TaskManager that offers this slot.
    +	 *
    +	 * @return The location info of the TaskManager that offers this slot
    +	 */
    +	TaskManagerLocation getTaskManagerLocation();
    +
    +	/**
    +	 * Gets the number of the slot.
    +	 *
    +	 * @return The number of the slot on the TaskManager.
    +	 */
    +	int getPhysicalSlotNumber();
    +
    +	/**
    +	 * Gets the actor gateway that can be used to send messages to the TaskManager.
    +	 * <p>
    +	 * This method should be removed once the new interface-based RPC abstraction is in place
    +	 *
    +	 * @return The actor gateway that can be used to send messages to the TaskManager.
    --- End diff --
    
    The fact that `TaskManagerGateway` can be an *actor gateway* is not something that is relevant for the Javadoc.


---

[GitHub] flink issue #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:

    https://github.com/apache/flink/pull/5088
  
    Thanks for the review @GJL. Merging this PR.


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r156879027
  
    --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/instance/SlotsTestImplContext.java ---
    @@ -31,7 +30,7 @@
     import static org.mockito.Mockito.mock;
     import static org.mockito.Mockito.when;
     
    -public class AllocatedSlotsTest {
    +public class SlotsTestImplContext {
    --- End diff --
    
    This is a mistake. Will correct it.


---

[GitHub] flink pull request #5088: [FLINK-8087] Decouple Slot from AllocatedSlot

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5088#discussion_r156880342
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/instance/AllocatedSlot.java ---
    @@ -144,6 +144,78 @@ public TaskManagerGateway getTaskManagerGateway() {
     		return taskManagerGateway;
     	}
     
    +	/**
    +	 * Triggers the release of the logical slot.
    +	 */
    +	public void triggerLogicalSlotRelease() {
    +		final LogicalSlot logicalSlot = logicalSlotReference.get();
    +
    +		if (logicalSlot != null) {
    +			logicalSlot.releaseSlot();
    +		}
    +	}
    +
    +	/**
    +	 * Releases the logical slot.
    +	 *
    +	 * @return true if the logical slot could be released, false otherwise.
    +	 */
    +	public boolean releaseLogicalSlot() {
    +		final LogicalSlot logicalSlot = logicalSlotReference.get();
    +
    +		if (logicalSlot != null) {
    +			if (logicalSlot instanceof Slot) {
    +				final Slot slot = (Slot) logicalSlot;
    +				if (slot.markReleased()) {
    +					logicalSlotReference.set(null);
    +					return true;
    +				}
    +			} else {
    +				throw new RuntimeException("Unsupported logical slot type encounterd " + logicalSlot.getClass());
    --- End diff --
    
    Good catch. Will correct it.


---