You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by zhijiangW <gi...@git.apache.org> on 2018/01/30 07:25:23 UTC
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
GitHub user zhijiangW opened a pull request:
https://github.com/apache/flink/pull/5381
[FLINK-8523][network] Stop assigning floating buffers for blocked input channels in exactly-once mode
## What is the purpose of the change
In exactly-once mode, the input channel is set blocked state when reading barrier from it. And the blocked state will be released after barrier alignment or cancelled.
In credit-based network flow control, we should avoid assigning floating buffers for blocked input channels because the buffers after barrier will not be processed by operator until alignment.
To do so, we can fully make use of floating buffers and speed up barrier alignment in some extent.
## Brief change log
- *Add `blockInputChannel` and `releaseBlockedInputChannels` in `InputGate` interface*
- *`UnionInputGate` constructs the mapping from channel index to `InputGate`*
- *`SingleInputGate` constructs the mapping from channel index to `InputChannel`*
- *`BarrierBuffer` determines the logic of blocking input channel or releasing it*
- *Avoid assigning floating buffers for blocked input channels*
## Verifying this change
This change added tests and can be verified as follows:
- *Added unit tests that validates the floating buffers are not assigned for blocked input channels*
- *Added unit test that validates the `BarrierBuffer` blocks or releases the `InputChannel` correctly*
- *Added unit test that validates the `UnionInputGate` and `SingleInputGate` constructs the mapping correctly*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhijiangW/flink FLINK-8523
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5381.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5381
----
commit 8c7ef5a22c348c3a6eedb04708b00093ae666e37
Author: Zhijiang <wa...@...>
Date: 2018-01-30T07:14:20Z
[FLINK-8523][network] Stop assigning floating buffers for blocked input channels in exactly-once mode
----
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164974094
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java ---
@@ -360,8 +360,9 @@ public boolean notifyBufferAvailable(Buffer buffer) {
// Important: double check the isReleased state inside synchronized block, so there is no
// race condition when notifyBufferAvailable and releaseAllResources running in parallel.
- if (isReleased.get() || bufferQueue.getAvailableBufferSize() >= numRequiredBuffers) {
+ if (isReleased.get() || isBlocked() || bufferQueue.getAvailableBufferSize() >= numRequiredBuffers) {
--- End diff --
shouldn't we accumulate/remember the count of "blocked" credits and restore/reassign them upon unblocking the channel? Doesn't the current code "leak" the credit? Or is there some other code, that will restore the credits balance (both exclusive/floating point buffers).
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164972989
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java ---
@@ -133,8 +134,13 @@
* Input channels. There is a one input channel for each consumed intermediate result partition.
* We store this in a map for runtime updates of single channels.
*/
+ @GuardedBy("requestLock")
private final Map<IntermediateResultPartitionID, InputChannel> inputChannels;
+ /** A mapping from internal channel index in this gate to input channel. */
+ @GuardedBy("requestLock")
+ private final Map<Integer, InputChannel> indexToInputChannelMap;
--- End diff --
nit: maybe rename to `channelIndexToInputChannel` and drop the comment and maybe drop the comment? It doesn't add much value.
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164979541
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/UnionInputGate.java ---
@@ -138,6 +148,24 @@ public void requestPartitions() throws IOException, InterruptedException {
}
}
+ @Override
+ public void blockInputChannel(int channelIndex) {
+ InputGate inputGate = indexToInputGateMap.get(channelIndex);
+ if (inputGate == null) {
+ throw new IllegalStateException("Could not find input gate from the channel index " + channelIndex);
+ }
+
+ int indexOffset = inputGateToIndexOffsetMap.get(inputGate);
+ inputGate.blockInputChannel(channelIndex - indexOffset);
--- End diff --
wrap this offset handling in a private method:
```
inputGate.blockInputChannel(externalToInternalChannelIndex(channelIndex, inputGate));
```
and similar call in `getNextBufferOrEvent()` method to:
```
bufferOrEvent.setChannelIndex(internalToExternalChannelIndex(bufferOrEvent.getChannelIndex(), inputGate));
```
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164974663
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputChannel.java ---
@@ -66,6 +66,9 @@
/** The current backoff (in ms) */
private int currentBackoff;
+ /** Flag indicating whether this channel is currently blocked or not. */
+ private volatile boolean isBlocked = false;
--- End diff --
As for now, blocking is not yet implemented/handled for `LocalInputChannel`s?
---
[GitHub] flink issue #5381: [FLINK-8523][network] Stop assigning floating buffers for...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/5381
Yes, I think it can make better use of floating buffers.
As last confirmation with Piotr, this PR will not be merged into release-1.5, because there were still some issues not confirmed before. So I pended this PR and did not update it temporarily.
What do you think of this PR issue? I can continue on it if necessary.
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164993308
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java ---
@@ -485,6 +494,23 @@ public void requestPartitions() throws IOException, InterruptedException {
}
}
+ @Override
+ public void blockInputChannel(int channelIndex) {
+ InputChannel inputChannel = indexToInputChannelMap.get(channelIndex);
+ if (inputChannel == null) {
+ throw new IllegalStateException("Could not find input channel from the channel index " + channelIndex);
--- End diff --
I referred to the other similar usage in `IllegalStateException`. If existing this condition that `channelIndex` is correct but the mapping construction is wrong.
I think `IllegalArgumentException` also makes sense.
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164987333
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputChannel.java ---
@@ -165,6 +168,14 @@ protected void notifyChannelNonEmpty() {
*/
abstract void releaseAllResources() throws IOException;
+ protected boolean isBlocked() {
--- End diff --
yes, we can define the abstract method here and implement in specific `RemoteInputChannel` by synchronization.
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164988413
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java ---
@@ -133,8 +134,13 @@
* Input channels. There is a one input channel for each consumed intermediate result partition.
* We store this in a map for runtime updates of single channels.
*/
+ @GuardedBy("requestLock")
private final Map<IntermediateResultPartitionID, InputChannel> inputChannels;
+ /** A mapping from internal channel index in this gate to input channel. */
+ @GuardedBy("requestLock")
+ private final Map<Integer, InputChannel> indexToInputChannelMap;
--- End diff --
ok
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164975471
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputChannel.java ---
@@ -165,6 +168,14 @@ protected void notifyChannelNonEmpty() {
*/
abstract void releaseAllResources() throws IOException;
+ protected boolean isBlocked() {
--- End diff --
If you move implementation of `isBlocked`/`setBlocked` methods to the `RemoteInputChannel` synchronisation can be optimised - instead of using another synchronisation method in form of `volatile`, `RemoteInputChannel` could use already existing `synchronized (bufferQueue)` for this purpose.
As it is now, this `volatile` `isBlocked` adds an additional overhead which could be avoided.
---
[GitHub] flink issue #5381: [FLINK-8523][network] Stop assigning floating buffers for...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/5381
@pnowojski I have submitted the whole process in one commit.
After you verify the implementation is feasible, I will submit a separate commit for adding unit tests based on this process.
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164990652
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/UnionInputGate.java ---
@@ -138,6 +148,24 @@ public void requestPartitions() throws IOException, InterruptedException {
}
}
+ @Override
+ public void blockInputChannel(int channelIndex) {
+ InputGate inputGate = indexToInputGateMap.get(channelIndex);
+ if (inputGate == null) {
+ throw new IllegalStateException("Could not find input gate from the channel index " + channelIndex);
+ }
+
+ int indexOffset = inputGateToIndexOffsetMap.get(inputGate);
+ inputGate.blockInputChannel(channelIndex - indexOffset);
--- End diff --
alright
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164975820
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java ---
@@ -315,6 +322,7 @@ public void returnExclusiveSegments(List<MemorySegment> segments) throws IOExcep
public void setInputChannel(IntermediateResultPartitionID partitionId, InputChannel inputChannel) {
synchronized (requestLock) {
if (inputChannels.put(checkNotNull(partitionId), checkNotNull(inputChannel)) == null
+ && indexToInputChannelMap.put(inputChannel.getChannelIndex(), inputChannel) == null
--- End diff --
Please extract `inputChannels.put(...)` and `indexToInputChannelMap.put(...)` to separate method wrapping those calls - this logic is duplicated with `updateInputChannel(...)`.
Also change it to something like this:
```
private boolean addInputChannel(inputChannel, partitionId) {
boolean alreadyInInputChannels = inputChannels.put(...) != null;
boolean alreadyInIndexesMapping = indexToInputChannelMap.put(...) != null;
checkState(alreadyInInputChannels ^ alreadyInIndexesMapping, "Inconsistent internal state (...)");
return alreadyInInputChannels;
}
```
---
[GitHub] flink issue #5381: [FLINK-8523][network] Stop assigning floating buffers for...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5381
@zhijiangW is this PR still useful? if so, was there any progress?
---
[GitHub] flink issue #5381: [FLINK-8523][network] Stop assigning floating buffers for...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/5381
There is one issue I am thinking.
After the input channel is unblocked, we can check whether this input channel needs request floating buffer from pool (if `availableNum < numRequiredBuffers` and `!isWaitingForFloatingBuffers`). Otherwise the request will be triggered by `onSenderBacklog` next time.
What do you think is better?
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164986882
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputChannel.java ---
@@ -66,6 +66,9 @@
/** The current backoff (in ms) */
private int currentBackoff;
+ /** Flag indicating whether this channel is currently blocked or not. */
+ private volatile boolean isBlocked = false;
--- End diff --
Yes, we have not improved the logic based on blocking state in `LocalInputChannel`. And it has the possibility to do that in future.
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164977052
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java ---
@@ -485,6 +494,23 @@ public void requestPartitions() throws IOException, InterruptedException {
}
}
+ @Override
+ public void blockInputChannel(int channelIndex) {
+ InputChannel inputChannel = indexToInputChannelMap.get(channelIndex);
+ if (inputChannel == null) {
+ throw new IllegalStateException("Could not find input channel from the channel index " + channelIndex);
--- End diff --
`IllegalArgumentException`?
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164988630
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java ---
@@ -315,6 +322,7 @@ public void returnExclusiveSegments(List<MemorySegment> segments) throws IOExcep
public void setInputChannel(IntermediateResultPartitionID partitionId, InputChannel inputChannel) {
synchronized (requestLock) {
if (inputChannels.put(checkNotNull(partitionId), checkNotNull(inputChannel)) == null
+ && indexToInputChannelMap.put(inputChannel.getChannelIndex(), inputChannel) == null
--- End diff --
alright.
---
[GitHub] flink pull request #5381: [FLINK-8523][network] Stop assigning floating buff...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on a diff in the pull request:
https://github.com/apache/flink/pull/5381#discussion_r164988217
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java ---
@@ -360,8 +360,9 @@ public boolean notifyBufferAvailable(Buffer buffer) {
// Important: double check the isReleased state inside synchronized block, so there is no
// race condition when notifyBufferAvailable and releaseAllResources running in parallel.
- if (isReleased.get() || bufferQueue.getAvailableBufferSize() >= numRequiredBuffers) {
+ if (isReleased.get() || isBlocked() || bufferQueue.getAvailableBufferSize() >= numRequiredBuffers) {
--- End diff --
That is the exact issue I pointed above.
I think it will not cause credit leak currently. But for further improvement, we can compare the current available credits with backlog after unblocking the channel. If not enough, we can trigger to request floating buffers at once. In current implementation, it has to wait next `onSenderBacklog` to trigger this process.
---
[GitHub] flink issue #5381: [FLINK-8523][network] Stop assigning floating buffers for...
Posted by zhijiangW <gi...@git.apache.org>.
Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/5381
Thanks for reviews and suggestions! :)
I will add some unit tests first to verify the related logics. For itcase, I will consider the necessary and feasibility.
---