You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by gu...@apache.org on 2024/06/24 07:42:10 UTC
(flink) branch master updated (2ad6f1e558e -> 67d23fc45be)
This is an automated email from the ASF dual-hosted git repository.
guoweijie pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
from 2ad6f1e558e [hotfix][runtime] Fix the issue of registering and unregistering a job for shuffle master
new 02032e2efb1 Revert "[FLINK-33668][runtime] Avoid the potential deadlock during reserving segments"
new 99ad724f664 Revert "[FLINK-33668][runtime] Close the producer client in case of the result partition is not finished properly"
new af1c8605d4c Revert "[FLINK-33668][runtime] Make the SortBuffer be able to return unused memory segments"
new 64c54e8776a Revert "[FLINK-33668][runtime] Decouple the network memory and job topology on result partition side"
new eaef742964a Revert "[FLINK-33668][runtime] Decouple the network memory and job topology on input gate side"
new 67d23fc45be Revert "[FLINK-33668][runtime] Redefine the redistribution logic of the NetworkBufferPool"
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../generated/all_taskmanager_network_section.html | 6 -
.../netty_shuffle_environment_configuration.html | 6 -
.../NettyShuffleEnvironmentOptions.java | 13 -
.../io/network/NettyShuffleServiceFactory.java | 1 -
.../runtime/io/network/buffer/BufferPool.java | 8 +-
.../io/network/buffer/BufferPoolFactory.java | 18 +-
.../runtime/io/network/buffer/LocalBufferPool.java | 105 ++---
.../io/network/buffer/NetworkBufferPool.java | 205 +++------
.../metrics/CreditBasedInputBuffersUsageGauge.java | 37 +-
.../metrics/ExclusiveBuffersUsageGauge.java | 2 +-
.../network/metrics/FloatingBuffersUsageGauge.java | 40 +-
.../network/metrics/NettyShuffleMetricFactory.java | 10 +-
.../network/netty/NettyPartitionRequestClient.java | 2 +-
.../partition/BufferWritingResultPartition.java | 2 +-
.../network/partition/ResultPartitionFactory.java | 17 +-
.../runtime/io/network/partition/SortBuffer.java | 27 +-
.../partition/SortMergeResultPartition.java | 2 +-
.../network/partition/consumer/BufferManager.java | 162 +------
.../partition/consumer/GateBuffersSpec.java | 34 +-
.../partition/consumer/InputGateSpecUtils.java | 63 +--
.../consumer/LocalRecoveredInputChannel.java | 15 -
.../partition/consumer/RecoveredInputChannel.java | 10 +-
.../partition/consumer/RemoteInputChannel.java | 30 +-
.../consumer/RemoteRecoveredInputChannel.java | 1 -
.../partition/consumer/SingleInputGate.java | 13 -
.../partition/consumer/SingleInputGateFactory.java | 53 +--
.../tiered/shuffle/TieredResultPartition.java | 4 -
.../shuffle/TieredResultPartitionFactory.java | 7 +-
.../tiered/storage/SortBufferAccumulator.java | 82 +---
.../tiered/storage/TieredStorageMemoryManager.java | 3 -
.../storage/TieredStorageMemoryManagerImpl.java | 63 ++-
.../flink/runtime/shuffle/NettyShuffleUtils.java | 45 +-
.../NettyShuffleEnvironmentConfiguration.java | 13 -
.../InputChannelRecoveredStateHandlerTest.java | 6 +-
.../SequentialChannelStateReaderImplTest.java | 2 +-
.../io/network/NettyShuffleEnvironmentBuilder.java | 9 -
.../io/network/NettyShuffleEnvironmentTest.java | 21 +-
.../api/writer/RecordWriterDelegateTest.java | 2 +-
.../io/network/api/writer/RecordWriterTest.java | 2 +-
.../io/network/buffer/BufferPoolFactoryTest.java | 186 ++------
.../io/network/buffer/LocalBufferPoolTest.java | 37 +-
.../io/network/buffer/NetworkBufferPoolTest.java | 38 +-
...UnpooledBufferPool.java => NoOpBufferPool.java} | 58 ++-
.../io/network/buffer/TestingBufferPool.java | 505 ---------------------
.../io/network/buffer/UnpooledBufferPool.java | 7 +-
...editBasedPartitionRequestClientHandlerTest.java | 34 +-
.../NettyMessageClientDecoderDelegateTest.java | 8 +-
.../NettyMessageClientSideSerializationTest.java | 8 +-
.../netty/NettyPartitionRequestClientTest.java | 8 +-
.../io/network/partition/DataBufferTest.java | 6 +-
.../network/partition/InputGateFairnessTest.java | 15 +-
.../io/network/partition/PartitionTestUtils.java | 16 -
.../PipelinedSubpartitionWithReadViewTest.java | 3 -
.../network/partition/ResultPartitionBuilder.java | 10 +-
.../partition/ResultPartitionFactoryTest.java | 1 -
.../io/network/partition/ResultPartitionTest.java | 10 +-
.../partition/SortMergeResultPartitionTest.java | 31 +-
.../partition/consumer/GateBuffersSpecTest.java | 116 ++---
.../consumer/InputBuffersMetricsTest.java | 36 +-
.../partition/consumer/LocalInputChannelTest.java | 4 +-
.../partition/consumer/RemoteInputChannelTest.java | 103 ++---
.../partition/consumer/SingleInputGateBuilder.java | 20 +-
.../partition/consumer/SingleInputGateTest.java | 56 +--
.../partition/hybrid/HsMemoryDataManagerTest.java | 8 +-
.../partition/hybrid/HsResultPartitionTest.java | 26 +-
.../partition/hybrid/HsSubpartitionViewTest.java | 2 +-
.../tiered/TestingTieredStorageMemoryManager.java | 20 -
.../tiered/shuffle/TieredResultPartitionTest.java | 29 +-
.../tiered/storage/HashBufferAccumulatorTest.java | 3 +-
.../tiered/storage/SortBufferAccumulatorTest.java | 18 +-
.../TieredStorageMemoryManagerImplTest.java | 13 +-
.../storage/TieredStorageSortBufferTest.java | 37 +-
.../runtime/shuffle/NettyShuffleUtilsTest.java | 12 +-
.../runtime/state/ChannelPersistenceITCase.java | 3 +-
.../AlignedCheckpointsMassiveRandomTest.java | 4 +-
.../checkpointing/CheckpointedInputGateTest.java | 6 +-
.../util/TestCheckpointedInputGateBuilder.java | 3 +-
.../flink/test/runtime/BatchShuffleITCaseBase.java | 22 +-
.../HybridShuffleMemoryDecouplingITCase.java | 114 -----
79 files changed, 669 insertions(+), 2108 deletions(-)
copy flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/{UnpooledBufferPool.java => NoOpBufferPool.java} (72%)
delete mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java
delete mode 100644 flink-tests/src/test/java/org/apache/flink/test/runtime/HybridShuffleMemoryDecouplingITCase.java
(flink) 01/06: Revert "[FLINK-33668][runtime] Avoid the potential deadlock during reserving segments"
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
guoweijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 02032e2efb138b9e3c588337714e6c2f64ff2499
Author: Yuxin Tan <ta...@gmail.com>
AuthorDate: Fri Jun 21 13:45:12 2024 +0800
Revert "[FLINK-33668][runtime] Avoid the potential deadlock during reserving segments"
This reverts commit 4171b980717e5dc313d781210a5bbfd35449ee50.
---
.../flink/runtime/io/network/buffer/LocalBufferPool.java | 15 ++++-----------
1 file changed, 4 insertions(+), 11 deletions(-)
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
index 99db252f7e4..c74d1a8e314 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
@@ -31,7 +31,6 @@ import javax.annotation.concurrent.GuardedBy;
import java.io.IOException;
import java.util.ArrayDeque;
-import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
@@ -273,22 +272,16 @@ public class LocalBufferPool implements BufferPool {
"Can not reserve more segments than number of minimum segments.");
CompletableFuture<?> toNotify = null;
-
- int numSegmentsNeeded;
synchronized (availableMemorySegments) {
checkDestroyed();
- numSegmentsNeeded = numberOfSegmentsToReserve - numberOfRequestedMemorySegments;
- }
- if (numSegmentsNeeded > 0) {
- List<MemorySegment> segments =
- networkBufferPool.requestPooledMemorySegmentsBlocking(numSegmentsNeeded);
- synchronized (availableMemorySegments) {
- availableMemorySegments.addAll(segments);
+ if (numberOfRequestedMemorySegments < numberOfSegmentsToReserve) {
+ availableMemorySegments.addAll(
+ networkBufferPool.requestPooledMemorySegmentsBlocking(
+ numberOfSegmentsToReserve - numberOfRequestedMemorySegments));
toNotify = availabilityHelper.getUnavailableToResetAvailable();
}
}
-
mayNotifyAvailable(toNotify);
}
(flink) 04/06: Revert "[FLINK-33668][runtime] Decouple the network memory and job topology on result partition side"
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
guoweijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 64c54e8776a3addb7ed9ba103ccadebce1a6c5e4
Author: Yuxin Tan <ta...@gmail.com>
AuthorDate: Fri Jun 21 14:00:06 2024 +0800
Revert "[FLINK-33668][runtime] Decouple the network memory and job topology on result partition side"
This reverts commit 88138d08e731d3084d59bf14cc2fb51bdf4afbd8.
---
.../generated/all_taskmanager_network_section.html | 6 -
.../netty_shuffle_environment_configuration.html | 6 -
.../NettyShuffleEnvironmentOptions.java | 13 -
.../io/network/NettyShuffleServiceFactory.java | 1 -
.../network/partition/ResultPartitionFactory.java | 8 -
.../runtime/io/network/partition/SortBuffer.java | 4 +-
.../network/partition/consumer/BufferManager.java | 2 +-
.../partition/consumer/InputGateSpecUtils.java | 8 +-
.../partition/consumer/SingleInputGateFactory.java | 6 +-
.../shuffle/TieredResultPartitionFactory.java | 7 +-
.../tiered/storage/SortBufferAccumulator.java | 68 +---
.../tiered/storage/TieredStorageMemoryManager.java | 3 -
.../storage/TieredStorageMemoryManagerImpl.java | 63 ++--
.../flink/runtime/shuffle/NettyShuffleUtils.java | 37 +-
.../NettyShuffleEnvironmentConfiguration.java | 13 -
.../io/network/NettyShuffleEnvironmentBuilder.java | 9 -
.../io/network/buffer/TestingBufferPool.java | 419 ++-------------------
...editBasedPartitionRequestClientHandlerTest.java | 7 +-
.../NettyMessageClientDecoderDelegateTest.java | 6 +-
.../NettyMessageClientSideSerializationTest.java | 6 +-
.../network/partition/InputGateFairnessTest.java | 9 +-
.../network/partition/ResultPartitionBuilder.java | 8 -
.../partition/ResultPartitionFactoryTest.java | 1 -
.../partition/consumer/GateBuffersSpecTest.java | 49 +--
.../partition/consumer/RemoteInputChannelTest.java | 21 +-
.../partition/consumer/SingleInputGateBuilder.java | 9 +-
.../tiered/TestingTieredStorageMemoryManager.java | 20 -
.../tiered/shuffle/TieredResultPartitionTest.java | 10 +-
.../tiered/storage/SortBufferAccumulatorTest.java | 15 +-
.../flink/test/runtime/BatchShuffleITCaseBase.java | 22 +-
.../HybridShuffleMemoryDecouplingITCase.java | 114 ------
31 files changed, 126 insertions(+), 844 deletions(-)
diff --git a/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html b/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html
index b0b957154ca..14e7532b050 100644
--- a/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html
+++ b/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html
@@ -26,12 +26,6 @@
<td>String</td>
<td>The option configures the class that is responsible for creating an external remote tier factory for hybrid shuffle. If configured, the hybrid shuffle will only initialize the specified remote tier according to the given class name. Currently, since the tier interfaces are not yet public and are still actively evolving, it is recommended that users do not independently implement the external remote tier until the tier interfaces are stabilized. </td>
</tr>
- <tr>
- <td><h5>taskmanager.network.hybrid-shuffle.memory-decoupling.enabled</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>The option is used to make the memory that is used by hybrid shuffle decoupled from the complexity of the job topology and the number of tasks on the task manager. It significantly reduces the chance of the "Insufficient number of network buffers" exception, while the workloads may suffer performance reduction silently.</td>
- </tr>
<tr>
<td><h5>taskmanager.network.hybrid-shuffle.remote.path</h5></td>
<td style="word-wrap: break-word;">(none)</td>
diff --git a/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html b/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html
index 1d3d33779a2..cfd54b05540 100644
--- a/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html
+++ b/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html
@@ -44,12 +44,6 @@
<td>String</td>
<td>The option configures the class that is responsible for creating an external remote tier factory for hybrid shuffle. If configured, the hybrid shuffle will only initialize the specified remote tier according to the given class name. Currently, since the tier interfaces are not yet public and are still actively evolving, it is recommended that users do not independently implement the external remote tier until the tier interfaces are stabilized. </td>
</tr>
- <tr>
- <td><h5>taskmanager.network.hybrid-shuffle.memory-decoupling.enabled</h5></td>
- <td style="word-wrap: break-word;">false</td>
- <td>Boolean</td>
- <td>The option is used to make the memory that is used by hybrid shuffle decoupled from the complexity of the job topology and the number of tasks on the task manager. It significantly reduces the chance of the "Insufficient number of network buffers" exception, while the workloads may suffer performance reduction silently.</td>
- </tr>
<tr>
<td><h5>taskmanager.network.hybrid-shuffle.remote.path</h5></td>
<td style="word-wrap: break-word;">(none)</td>
diff --git a/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java b/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java
index 3f5c2b2e5cd..72cf1404cb4 100644
--- a/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java
+++ b/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java
@@ -525,19 +525,6 @@ public class NettyShuffleEnvironmentOptions {
+ HYBRID_SHUFFLE_NEW_MODE_OPTION_NAME
+ " is true, the remote storage will be disabled.");
- /** The option to decouple the needed memory of hybrid shuffle and the job topology. */
- @Documentation.Section(Documentation.Sections.ALL_TASK_MANAGER_NETWORK)
- @Experimental
- public static final ConfigOption<Boolean> NETWORK_HYBRID_SHUFFLE_ENABLE_MEMORY_DECOUPLING =
- key("taskmanager.network.hybrid-shuffle.memory-decoupling.enabled")
- .booleanType()
- .defaultValue(false)
- .withDescription(
- "The option is used to make the memory that is used by hybrid shuffle decoupled from the complexity of the"
- + " job topology and the number of tasks on the task manager. It significantly reduces the chance of "
- + "the \"Insufficient number of network buffers\" exception, while the workloads may suffer performance "
- + "reduction silently.");
-
/**
* @deprecated The hash-based blocking shuffle is deprecated in 1.20 and will be totally removed
* in 2.0.
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleServiceFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleServiceFactory.java
index f2e65d7317c..d8d068d9aee 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleServiceFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleServiceFactory.java
@@ -238,7 +238,6 @@ public class NettyShuffleServiceFactory
config.getMaxOverdraftBuffersPerGate(),
config.getHybridShuffleSpilledIndexRegionGroupSize(),
config.getHybridShuffleNumRetainedInMemoryRegionsMax(),
- config.isMemoryDecouplingEnabled(),
tieredResultPartitionFactory);
SingleInputGateFactory singleInputGateFactory =
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
index fa100455708..1bf21f8c7bc 100755
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
@@ -30,7 +30,6 @@ import org.apache.flink.runtime.io.network.buffer.BufferPool;
import org.apache.flink.runtime.io.network.buffer.BufferPoolFactory;
import org.apache.flink.runtime.io.network.partition.hybrid.HsResultPartition;
import org.apache.flink.runtime.io.network.partition.hybrid.HybridShuffleConfiguration;
-import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils;
import org.apache.flink.runtime.io.network.partition.hybrid.tiered.shuffle.TieredResultPartitionFactory;
import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageMemorySpec;
import org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.TierFactory;
@@ -90,8 +89,6 @@ public class ResultPartitionFactory {
private final long hybridShuffleNumRetainedInMemoryRegionsMax;
- private final boolean memoryDecouplingEnabled;
-
private final boolean sslEnabled;
private final int maxOverdraftBuffersPerGate;
@@ -117,7 +114,6 @@ public class ResultPartitionFactory {
int maxOverdraftBuffersPerGate,
int hybridShuffleSpilledIndexRegionGroupSize,
long hybridShuffleNumRetainedInMemoryRegionsMax,
- boolean memoryDecouplingEnabled,
Optional<TieredResultPartitionFactory> tieredStorage) {
this.partitionManager = partitionManager;
@@ -139,7 +135,6 @@ public class ResultPartitionFactory {
this.hybridShuffleSpilledIndexRegionGroupSize = hybridShuffleSpilledIndexRegionGroupSize;
this.hybridShuffleNumRetainedInMemoryRegionsMax =
hybridShuffleNumRetainedInMemoryRegionsMax;
- this.memoryDecouplingEnabled = memoryDecouplingEnabled;
this.tieredStorage = tieredStorage;
}
@@ -272,7 +267,6 @@ public class ResultPartitionFactory {
maxParallelism,
networkBufferSize,
isBroadcast,
- memoryDecouplingEnabled,
partitionManager,
bufferCompressor,
checkNotNull(
@@ -397,11 +391,9 @@ public class ResultPartitionFactory {
sortShuffleMinBuffers,
numberOfSubpartitions,
tieredStorage.isPresent(),
- memoryDecouplingEnabled,
tieredStorage
.map(ResultPartitionFactory::getNumTotalGuaranteedBuffers)
.orElse(0),
- TieredStorageUtils.getMinBuffersPerResultPartition(),
type);
return bufferPoolFactory.createBufferPool(
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java
index 371484696a6..6bdd0eaa65a 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java
@@ -242,11 +242,11 @@ public abstract class SortBuffer implements DataBuffer {
}
// allocate exactly enough buffers for the appended record
- while (availableBytes < numBytesRequired) {
+ do {
MemorySegment segment = freeSegments.poll();
availableBytes += bufferSize;
addBuffer(checkNotNull(segment));
- }
+ } while (availableBytes < numBytesRequired);
return true;
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java
index 8c32655a6cb..9a39e82168a 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java
@@ -174,7 +174,7 @@ public class BufferManager implements BufferListener, BufferRecycler {
/** Requests exclusive buffers from the local buffer pool. */
void requestExclusiveBuffers() {
synchronized (bufferQueue) {
- checkState(numExclusiveBuffers >= 0, "Num exclusive buffers must be non-negative.");
+ checkArgument(numExclusiveBuffers >= 0, "Num exclusive buffers must be non-negative.");
resizeBufferQueue();
if (numExclusiveBuffers == 0) {
return;
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
index 59ec4c5a903..661843cd096 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
@@ -39,8 +39,7 @@ public class InputGateSpecUtils {
int configuredFloatingNetworkBuffersPerGate,
ResultPartitionType partitionType,
int numInputChannels,
- boolean enableTieredStorage,
- boolean memoryDecouplingEnabled) {
+ boolean enableTieredStorage) {
int maxRequiredBuffersThresholdPerGate =
getEffectiveMaxRequiredBuffersPerGate(
partitionType, configuredMaxRequiredBuffersPerGate, enableTieredStorage);
@@ -60,9 +59,7 @@ public class InputGateSpecUtils {
numInputChannels,
expectedBuffersPerGate);
int minBuffersPerGate =
- partitionType.isHybridResultPartition()
- && enableTieredStorage
- && memoryDecouplingEnabled
+ partitionType.isHybridResultPartition() && enableTieredStorage
? TieredStorageUtils.getMinBuffersPerGate()
: expectedBuffersPerGate;
expectedBuffersPerGate = Math.max(minBuffersPerGate, expectedBuffersPerGate);
@@ -115,6 +112,7 @@ public class InputGateSpecUtils {
return numInputChannels * configuredNetworkBuffersPerChannel + 1;
}
+ /** */
private static int getMaxBuffersPerGate(
int numInputChannels,
int configuredNetworkBuffersPerChannel,
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
index 72b49b0a6e3..35700a2c460 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
@@ -104,8 +104,6 @@ public class SingleInputGateFactory {
private final int networkBufferSize;
- private final boolean memoryDecouplingEnabled;
-
private final BufferDebloatConfiguration debloatConfiguration;
/** The following attributes will be null if tiered storage shuffle is disabled. */
@@ -134,7 +132,6 @@ public class SingleInputGateFactory {
this.batchShuffleCompressionEnabled = networkConfig.isBatchShuffleCompressionEnabled();
this.compressionCodec = networkConfig.getCompressionCodec();
this.networkBufferSize = networkConfig.networkBufferSize();
- this.memoryDecouplingEnabled = networkConfig.isMemoryDecouplingEnabled();
this.connectionManager = connectionManager;
this.partitionManager = partitionManager;
this.taskEventPublisher = taskEventPublisher;
@@ -167,8 +164,7 @@ public class SingleInputGateFactory {
floatingNetworkBuffersPerGate,
igdd.getConsumedPartitionType(),
calculateNumNonLocalChannels(igdd, isSharedInputChannelSupported),
- tieredStorageConfiguration != null,
- memoryDecouplingEnabled);
+ tieredStorageConfiguration != null);
SupplierWithException<BufferPool, IOException> bufferPoolFactory =
createBufferPoolFactory(
networkBufferPool,
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionFactory.java
index 8a788fe9258..4a00401c4d9 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionFactory.java
@@ -93,7 +93,6 @@ public class TieredResultPartitionFactory {
int maxParallelism,
int bufferSizeBytes,
Boolean isBroadCastOnly,
- boolean memoryDecouplingEnabled,
ResultPartitionManager partitionManager,
@Nullable BufferCompressor bufferCompressor,
List<TierShuffleDescriptor> tierShuffleDescriptors,
@@ -115,7 +114,6 @@ public class TieredResultPartitionFactory {
numSubpartitions,
bufferSizeBytes,
numAccumulatorExclusiveBuffers,
- memoryDecouplingEnabled,
memoryManager,
isNumberOfPartitionConsumerUndefined);
@@ -166,19 +164,16 @@ public class TieredResultPartitionFactory {
int numSubpartitions,
int bufferSizeBytes,
int numAccumulatorExclusiveBuffers,
- boolean enableMemoryDecoupling,
TieredStorageMemoryManager storageMemoryManager,
boolean isNumberOfPartitionConsumerUndefined) {
- long poolSizeCheckInterval = TieredStorageUtils.getPoolSizeCheckInterval();
BufferAccumulator bufferAccumulator;
- if (enableMemoryDecoupling || (numSubpartitions + 1) > numAccumulatorExclusiveBuffers) {
+ if ((numSubpartitions + 1) > numAccumulatorExclusiveBuffers) {
bufferAccumulator =
new SortBufferAccumulator(
numSubpartitions,
numAccumulatorExclusiveBuffers,
bufferSizeBytes,
- enableMemoryDecoupling ? poolSizeCheckInterval : 0,
storageMemoryManager,
!isNumberOfPartitionConsumerUndefined);
} else {
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java
index 8dc6eb2d69f..4c9d3cf7e79 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java
@@ -27,8 +27,6 @@ import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
import org.apache.flink.runtime.io.network.partition.BufferWithSubpartition;
import org.apache.flink.runtime.io.network.partition.DataBuffer;
import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
-import org.apache.flink.util.ExceptionUtils;
-import org.apache.flink.util.concurrent.ExecutorThreadFactory;
import org.apache.flink.util.function.TriConsumer;
import javax.annotation.Nullable;
@@ -37,11 +35,6 @@ import javax.annotation.concurrent.GuardedBy;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.LinkedList;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-import java.util.concurrent.atomic.AtomicInteger;
import static org.apache.flink.util.Preconditions.checkNotNull;
import static org.apache.flink.util.Preconditions.checkState;
@@ -54,9 +47,9 @@ import static org.apache.flink.util.Preconditions.checkState;
*
* <p>The {@link SortBufferAccumulator} can help use less buffers to accumulate data, which
* decouples the buffer usage with the number of parallelism. The number of buffers used by the
- * {@link SortBufferAccumulator} will be numExpectedBuffers at most. Once the {@link DataBuffer} is
- * full, or switching from broadcast to non-broadcast(or vice versa), the buffer in the sort buffer
- * will be flushed to the tiers.
+ * {@link SortBufferAccumulator} will be numBuffers at most. Once the {@link DataBuffer} is full, or
+ * switching from broadcast to non-broadcast(or vice versa), the buffer in the sort buffer will be
+ * flushed to the tiers.
*
* <p>Note that this class need not be thread-safe, because it should only be accessed from the main
* thread.
@@ -66,8 +59,8 @@ public class SortBufferAccumulator implements BufferAccumulator {
/** The number of the subpartitions. */
private final int numSubpartitions;
- /** The number of the expected buffers used by the {@link SortBufferAccumulator}. */
- private final int numExpectedBuffers;
+ /** The total number of the buffers used by the {@link SortBufferAccumulator}. */
+ private final int numBuffers;
/** The byte size of one single buffer. */
private final int bufferSizeBytes;
@@ -105,18 +98,6 @@ public class SortBufferAccumulator implements BufferAccumulator {
@Nullable
private TriConsumer<TieredStorageSubpartitionId, Buffer, Integer> accumulatedBufferFlusher;
- /**
- * An executor to periodically check the size of buffer pool. If the size is changed, the
- * accumulated buffers should be flushed to release the buffers.
- */
- private final ScheduledExecutorService periodicalAccumulatorFlusher =
- Executors.newSingleThreadScheduledExecutor(
- new ExecutorThreadFactory("hybrid-shuffle-periodical-accumulator-flusher"));
-
- private final long poolSizeCheckInterval;
-
- private AtomicInteger poolSize;
-
/** Whether the current {@link DataBuffer} is a broadcast sort buffer. */
private boolean isBroadcastDataBuffer;
@@ -127,37 +108,20 @@ public class SortBufferAccumulator implements BufferAccumulator {
public SortBufferAccumulator(
int numSubpartitions,
- int numExpectedBuffers,
+ int numBuffers,
int bufferSizeBytes,
- long poolSizeCheckInterval,
TieredStorageMemoryManager memoryManager,
boolean isPartialRecordAllowed) {
this.numSubpartitions = numSubpartitions;
this.bufferSizeBytes = bufferSizeBytes;
- this.numExpectedBuffers = numExpectedBuffers;
- this.poolSizeCheckInterval = poolSizeCheckInterval;
+ this.numBuffers = numBuffers;
this.memoryManager = memoryManager;
this.isPartialRecordAllowed = isPartialRecordAllowed;
- this.poolSize = new AtomicInteger(-1);
}
@Override
public void setup(TriConsumer<TieredStorageSubpartitionId, Buffer, Integer> bufferFlusher) {
this.accumulatedBufferFlusher = bufferFlusher;
-
- if (poolSizeCheckInterval > 0) {
- periodicalAccumulatorFlusher.scheduleWithFixedDelay(
- () -> {
- int newSize = this.memoryManager.getBufferPoolSize();
- int oldSize = poolSize.getAndSet(newSize);
- if (oldSize > newSize) {
- flushCurrentDataBuffer();
- }
- },
- poolSizeCheckInterval,
- poolSizeCheckInterval,
- TimeUnit.MILLISECONDS);
- }
}
@Override
@@ -192,17 +156,6 @@ public class SortBufferAccumulator implements BufferAccumulator {
@Override
public void close() {
- if (periodicalAccumulatorFlusher != null) {
- periodicalAccumulatorFlusher.shutdown();
- try {
- if (!periodicalAccumulatorFlusher.awaitTermination(5L, TimeUnit.MINUTES)) {
- throw new TimeoutException(
- "Timeout for shutting down the periodical accumulator flusher.");
- }
- } catch (Exception e) {
- ExceptionUtils.rethrow(e);
- }
- }
synchronized (lock) {
flushCurrentDataBuffer();
isDataBufferReleased = true;
@@ -248,8 +201,7 @@ public class SortBufferAccumulator implements BufferAccumulator {
@GuardedBy("lock")
private void requestBuffers() {
- while (freeSegments.size()
- < Math.min(numExpectedBuffers, memoryManager.getBufferPoolSize() - 1)) {
+ while (freeSegments.size() < numBuffers) {
Buffer buffer = requestBuffer();
freeSegments.add(checkNotNull(buffer).getMemorySegment());
if (bufferRecycler == null) {
@@ -267,7 +219,7 @@ public class SortBufferAccumulator implements BufferAccumulator {
}
currentDataBuffer.finish();
- while (currentDataBuffer.hasRemaining()) {
+ do {
MemorySegment freeSegment = getFreeSegment();
BufferWithSubpartition bufferWithSubpartition =
currentDataBuffer.getNextBuffer(freeSegment);
@@ -280,7 +232,7 @@ public class SortBufferAccumulator implements BufferAccumulator {
((double) currentDataBuffer.getRecordRemainingBytes())
/ bufferSizeBytes);
flushBuffer(bufferWithSubpartition, numRemainingConsecutiveBuffers);
- }
+ } while (true);
isDataBufferReleased = true;
releaseFreeBuffers();
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManager.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManager.java
index c2793b72bbb..5bf439064b5 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManager.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManager.java
@@ -146,9 +146,6 @@ public interface TieredStorageMemoryManager {
*/
void transferBufferOwnership(Object oldOwner, Object newOwner, Buffer buffer);
- /** @return the size of the local buffer pool or -1 if the buffer pool is not initialized. */
- int getBufferPoolSize();
-
/**
* Release all the resources(if exists) and check the state of the {@link
* TieredStorageMemoryManager}.
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImpl.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImpl.java
index 0ef453ec126..7d7aaf43f0d 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImpl.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImpl.java
@@ -210,16 +210,12 @@ public class TieredStorageMemoryManagerImpl implements TieredStorageMemoryManage
reclaimBuffersIfNeeded(0);
MemorySegment memorySegment = bufferQueue.poll();
- // Requests the buffer as soon as possible from wherever it is available in the buffer pool
- // or the internal queue.
- hardBackpressureTimerGauge.markStart();
- while (memorySegment == null) {
+ if (memorySegment == null) {
memorySegment = requestBufferBlockingFromPool();
- if (memorySegment == null) {
- memorySegment = requestBufferFromQueue();
- }
}
- hardBackpressureTimerGauge.markEnd();
+ if (memorySegment == null) {
+ memorySegment = checkNotNull(requestBufferBlockingFromQueue());
+ }
incNumRequestedBuffer(owner);
return new BufferBuilder(
@@ -261,9 +257,11 @@ public class TieredStorageMemoryManagerImpl implements TieredStorageMemoryManage
while (bufferQueue.size() + numRequestedByGuaranteedReclaimableOwners
< numGuaranteedReclaimableBuffers + numAdditionalBuffers) {
- hardBackpressureTimerGauge.markStart();
+ if (numRequestedBuffers.get() >= bufferPool.getNumBuffers()) {
+ return false;
+ }
+
MemorySegment memorySegment = requestBufferBlockingFromPool();
- hardBackpressureTimerGauge.markEnd();
if (memorySegment == null) {
return false;
}
@@ -290,30 +288,25 @@ public class TieredStorageMemoryManagerImpl implements TieredStorageMemoryManage
try {
releasedStateLock.writeLock().lock();
isReleased = true;
-
- if (executor != null) {
- executor.shutdown();
- try {
- if (!executor.awaitTermination(5L, TimeUnit.MINUTES)) {
- throw new TimeoutException(
- "Timeout for shutting down the buffer reclaim checker executor.");
- }
- } catch (Exception e) {
- ExceptionUtils.rethrow(e);
- }
- }
- while (!bufferQueue.isEmpty()) {
- MemorySegment segment = bufferQueue.poll();
- bufferPool.recycle(segment);
- numRequestedBuffers.decrementAndGet();
- }
} finally {
releasedStateLock.writeLock().unlock();
}
- }
-
- public int getBufferPoolSize() {
- return bufferPool == null ? -1 : bufferPool.getNumBuffers();
+ if (executor != null) {
+ executor.shutdown();
+ try {
+ if (!executor.awaitTermination(5L, TimeUnit.MINUTES)) {
+ throw new TimeoutException(
+ "Timeout for shutting down the buffer reclaim checker executor.");
+ }
+ } catch (Exception e) {
+ ExceptionUtils.rethrow(e);
+ }
+ }
+ while (!bufferQueue.isEmpty()) {
+ MemorySegment segment = bufferQueue.poll();
+ bufferPool.recycle(segment);
+ numRequestedBuffers.decrementAndGet();
+ }
}
/**
@@ -323,6 +316,8 @@ public class TieredStorageMemoryManagerImpl implements TieredStorageMemoryManage
@Nullable
private MemorySegment requestBufferBlockingFromPool() {
MemorySegment memorySegment = null;
+
+ hardBackpressureTimerGauge.markStart();
while (numRequestedBuffers.get() < bufferPool.getNumBuffers()) {
memorySegment = bufferPool.requestMemorySegment();
if (memorySegment == null) {
@@ -339,20 +334,20 @@ public class TieredStorageMemoryManagerImpl implements TieredStorageMemoryManage
break;
}
}
+ hardBackpressureTimerGauge.markEnd();
return memorySegment;
}
/** @return a memory segment from the internal buffer queue. */
- @Nullable
- private MemorySegment requestBufferFromQueue() {
+ private MemorySegment requestBufferBlockingFromQueue() {
CompletableFuture<Void> requestBufferFuture = new CompletableFuture<>();
scheduleCheckRequestBufferFuture(
requestBufferFuture, INITIAL_REQUEST_BUFFER_TIMEOUT_FOR_RECLAIMING_MS);
MemorySegment memorySegment = null;
try {
- memorySegment = bufferQueue.poll(100, TimeUnit.MILLISECONDS);
+ memorySegment = bufferQueue.take();
} catch (InterruptedException e) {
ExceptionUtils.rethrow(e);
} finally {
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java b/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java
index 262f7c476c9..73c54ce1ff4 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java
@@ -40,6 +40,10 @@ import static org.apache.flink.util.Preconditions.checkState;
*/
public class NettyShuffleUtils {
+ // Temporarily declare the default value here, it would be moved to the configuration class
+ // later.
+ private static final int DEFAULT_HYBRID_SHUFFLE_MIN_BUFFERS = 8;
+
/**
* Calculates and returns the number of required exclusive network buffers per input channel.
*/
@@ -69,32 +73,32 @@ public class NettyShuffleUtils {
final int sortShuffleMinBuffers,
final int numSubpartitions,
final boolean enableTieredStorage,
- final boolean enableMemoryDecoupling,
final int tieredStoreExclusiveBuffers,
- final int tieredStorageMinBuffersPerResultPartition,
final ResultPartitionType type) {
boolean isSortShuffle =
type.isBlockingOrBlockingPersistentResultPartition()
&& numSubpartitions >= sortShuffleMinParallelism;
- boolean isMemoryDecouplingEnabled =
- type.isHybridResultPartition() && enableTieredStorage && enableMemoryDecoupling;
-
- int expected =
- isSortShuffle
- ? sortShuffleMinBuffers
- : (enableTieredStorage
- ? Math.min(tieredStoreExclusiveBuffers, numSubpartitions + 1)
- : (numSubpartitions + 1));
+ int expected;
+ if (isSortShuffle) {
+ expected = sortShuffleMinBuffers;
+ } else {
+ expected =
+ enableTieredStorage
+ ? Math.min(tieredStoreExclusiveBuffers, numSubpartitions + 1)
+ : (numSubpartitions + 1);
+ }
- int min = isMemoryDecouplingEnabled ? tieredStorageMinBuffersPerResultPartition : expected;
- expected = Math.max(min, expected);
+ int min = expected;
+ if (type.isHybridResultPartition()) {
+ min = Math.min(DEFAULT_HYBRID_SHUFFLE_MIN_BUFFERS, expected);
+ }
int max =
type.isBounded()
? numSubpartitions * configuredNetworkBuffersPerChannel
+ numFloatingBuffersPerGate
: (isSortShuffle
- ? Math.max(min, 4 * numSubpartitions)
+ ? 4 * numSubpartitions
: NetworkBufferPool.UNBOUNDED_POOL_SIZE);
max = Math.max(max, expected);
// for each upstream hash-based blocking/pipelined subpartition, at least one buffer is
@@ -166,7 +170,6 @@ public class NettyShuffleUtils {
floatingNetworkBuffersPerGate,
type,
numInputChannels,
- false,
false);
return gateBuffersSpec.getMaxBuffersPerGate();
}
@@ -187,9 +190,7 @@ public class NettyShuffleUtils {
sortShuffleMinBuffers,
numSubpartitions,
false,
- false,
0,
- 1,
type);
// In order to avoid network buffer request timeout (see FLINK-12852), we announce
@@ -198,7 +199,7 @@ public class NettyShuffleUtils {
// back pressure so we need to include all the floating buffers in the announcement, i.e. we
// should take the max value;
// 2. For blocking shuffle, it is back pressure free and floating buffers can be recycled
- // in time, so that the expected buffers would be enough.
+ // in time, so that the minimum required buffers would be enough.
int ret = type.canBePipelinedConsumed() ? tuple.f2 : tuple.f0;
if (ret == Integer.MAX_VALUE) {
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/NettyShuffleEnvironmentConfiguration.java b/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/NettyShuffleEnvironmentConfiguration.java
index ae29a8d774f..ac4b15e383c 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/NettyShuffleEnvironmentConfiguration.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/NettyShuffleEnvironmentConfiguration.java
@@ -49,7 +49,6 @@ import java.util.Optional;
import static org.apache.flink.api.common.BatchShuffleMode.ALL_EXCHANGES_HYBRID_FULL;
import static org.apache.flink.api.common.BatchShuffleMode.ALL_EXCHANGES_HYBRID_SELECTIVE;
import static org.apache.flink.configuration.ExecutionOptions.BATCH_SHUFFLE_MODE;
-import static org.apache.flink.configuration.NettyShuffleEnvironmentOptions.NETWORK_HYBRID_SHUFFLE_ENABLE_MEMORY_DECOUPLING;
import static org.apache.flink.configuration.NettyShuffleEnvironmentOptions.NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE;
import static org.apache.flink.util.Preconditions.checkArgument;
@@ -118,8 +117,6 @@ public class NettyShuffleEnvironmentConfiguration {
private final long hybridShuffleNumRetainedInMemoryRegionsMax;
- private final boolean isMemoryDecouplingEnabled;
-
private final TieredStorageConfiguration tieredStorageConfiguration;
public NettyShuffleEnvironmentConfiguration(
@@ -148,7 +145,6 @@ public class NettyShuffleEnvironmentConfiguration {
int maxOverdraftBuffersPerGate,
int hybridShuffleSpilledIndexRegionGroupSize,
long hybridShuffleNumRetainedInMemoryRegionsMax,
- boolean isMemoryDecouplingEnabled,
@Nullable TieredStorageConfiguration tieredStorageConfiguration) {
this.numNetworkBuffers = numNetworkBuffers;
@@ -177,7 +173,6 @@ public class NettyShuffleEnvironmentConfiguration {
this.hybridShuffleSpilledIndexRegionGroupSize = hybridShuffleSpilledIndexRegionGroupSize;
this.hybridShuffleNumRetainedInMemoryRegionsMax =
hybridShuffleNumRetainedInMemoryRegionsMax;
- this.isMemoryDecouplingEnabled = isMemoryDecouplingEnabled;
this.tieredStorageConfiguration = tieredStorageConfiguration;
}
@@ -283,10 +278,6 @@ public class NettyShuffleEnvironmentConfiguration {
return hybridShuffleNumRetainedInMemoryRegionsMax;
}
- public boolean isMemoryDecouplingEnabled() {
- return isMemoryDecouplingEnabled;
- }
-
public int getHybridShuffleSpilledIndexRegionGroupSize() {
return hybridShuffleSpilledIndexRegionGroupSize;
}
@@ -419,9 +410,6 @@ public class NettyShuffleEnvironmentConfiguration {
NettyShuffleEnvironmentOptions
.HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX);
- boolean isMemoryDecouplingEnabled =
- configuration.get(NETWORK_HYBRID_SHUFFLE_ENABLE_MEMORY_DECOUPLING);
-
checkArgument(buffersPerChannel >= 0, "Must be non-negative.");
checkArgument(
!maxRequiredBuffersPerGate.isPresent() || maxRequiredBuffersPerGate.get() >= 1,
@@ -468,7 +456,6 @@ public class NettyShuffleEnvironmentConfiguration {
maxOverdraftBuffersPerGate,
hybridShuffleSpilledIndexSegmentSize,
hybridShuffleNumRetainedInMemoryRegionsMax,
- isMemoryDecouplingEnabled,
tieredStorageConfiguration);
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentBuilder.java
index 00596641441..3ffeafe204b 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentBuilder.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentBuilder.java
@@ -102,8 +102,6 @@ public class NettyShuffleEnvironmentBuilder {
private long hybridShuffleNumRetainedInMemoryRegionsMax = Long.MAX_VALUE;
- private boolean isMemoryDecouplingEnabled = false;
-
private int hybridShuffleSpilledIndexSegmentSize = 256;
private TieredStorageConfiguration tieredStorageConfiguration = null;
@@ -242,12 +240,6 @@ public class NettyShuffleEnvironmentBuilder {
return this;
}
- public NettyShuffleEnvironmentBuilder setIsMemoryDecouplingEnabled(
- boolean isMemoryDecouplingEnabled) {
- this.isMemoryDecouplingEnabled = isMemoryDecouplingEnabled;
- return this;
- }
-
public NettyShuffleEnvironmentBuilder setHybridShuffleSpilledIndexSegmentSize(
int hybridShuffleSpilledIndexSegmentSize) {
this.hybridShuffleSpilledIndexSegmentSize = hybridShuffleSpilledIndexSegmentSize;
@@ -288,7 +280,6 @@ public class NettyShuffleEnvironmentBuilder {
maxOverdraftBuffersPerGate,
hybridShuffleSpilledIndexSegmentSize,
hybridShuffleNumRetainedInMemoryRegionsMax,
- isMemoryDecouplingEnabled,
tieredStorageConfiguration),
taskManagerLocation,
new TaskEventDispatcher(),
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java
index c0c7bbc4e75..e34c4bb8dac 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java
@@ -21,485 +21,118 @@
package org.apache.flink.runtime.io.network.buffer;
import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
-import javax.annotation.Nullable;
-
-import java.io.IOException;
import java.util.concurrent.CompletableFuture;
-import java.util.function.Consumer;
-import java.util.function.Function;
-import java.util.function.Supplier;
/** Implementation of {@link BufferPool} for testing. */
public class TestingBufferPool implements BufferPool {
- public static final TestingBufferPool NO_OP = TestingBufferPool.builder().build();
-
- private Consumer<Integer> reserveSegmentsConsumer;
-
- private Runnable lazyDestroyRunnable;
-
- private Supplier<Buffer> requestBufferSupplier;
-
- private Supplier<BufferBuilder> requestBufferBuilderSupplier;
-
- private Supplier<BufferBuilder> requestBufferBuilderBlockingSupplier;
-
- private Function<Integer, BufferBuilder> requestBufferBuilderFunction;
-
- private Function<Integer, BufferBuilder> requestBufferBuilderBlockingFunction;
-
- private Function<BufferListener, Boolean> addBufferListenerFunction;
-
- private Supplier<Boolean> isDestroyedSupplier;
-
- private Supplier<MemorySegment> requestMemorySegmentSupplier;
-
- private Supplier<MemorySegment> requestMemorySegmentBlockingSupplier;
-
- private Supplier<Integer> getExpectedNumberOfMemorySegmentsSupplier;
-
- private Supplier<Integer> getMinNumberOfMemorySegmentsSupplier;
-
- private Supplier<Integer> getMaxNumberOfMemorySegmentsSupplier;
-
- private Supplier<Integer> getNumBuffersSupplier;
-
- private Consumer<Integer> setNumBuffersConsumer;
-
- private Consumer<Integer> setMaxOverdraftBuffersPerGateConsumer;
-
- private Supplier<Integer> getMaxOverdraftBuffersPerGateSupplier;
-
- private Supplier<Integer> getNumberOfAvailableMemorySegmentsSupplier;
-
- private Supplier<Integer> bestEffortGetNumOfUsedBuffersSupplier;
-
- private Consumer<MemorySegment> recycleConsumer;
-
- private Supplier<CompletableFuture<?>> getAvailableFutureSupplier;
-
- private TestingBufferPool(
- Consumer<Integer> reserveSegmentsConsumer,
- Runnable lazyDestroyRunnable,
- Supplier<Buffer> requestBufferSupplier,
- Supplier<BufferBuilder> requestBufferBuilderSupplier,
- Supplier<BufferBuilder> requestBufferBuilderBlockingSupplier,
- Function<Integer, BufferBuilder> requestBufferBuilderFunction,
- Function<Integer, BufferBuilder> requestBufferBuilderBlockingFunction,
- Function<BufferListener, Boolean> addBufferListenerFunction,
- Supplier<Boolean> isDestroyedSupplier,
- Supplier<MemorySegment> requestMemorySegmentSupplier,
- Supplier<MemorySegment> requestMemorySegmentBlockingSupplier,
- Supplier<Integer> getExpectedNumberOfMemorySegmentsSupplier,
- Supplier<Integer> getMinNumberOfMemorySegmentsSupplier,
- Supplier<Integer> getMaxNumberOfMemorySegmentsSupplier,
- Supplier<Integer> getNumBuffersSupplier,
- Consumer<Integer> setNumBuffersConsumer,
- Consumer<Integer> setMaxOverdraftBuffersPerGateConsumer,
- Supplier<Integer> getMaxOverdraftBuffersPerGateSupplier,
- Supplier<Integer> getNumberOfAvailableMemorySegmentsSupplier,
- Supplier<Integer> bestEffortGetNumOfUsedBuffersSupplier,
- Consumer<MemorySegment> recycleConsumer,
- Supplier<CompletableFuture<?>> getAvailableFutureSupplier) {
- this.reserveSegmentsConsumer = reserveSegmentsConsumer;
- this.lazyDestroyRunnable = lazyDestroyRunnable;
- this.requestBufferSupplier = requestBufferSupplier;
- this.requestBufferBuilderSupplier = requestBufferBuilderSupplier;
- this.requestBufferBuilderBlockingSupplier = requestBufferBuilderBlockingSupplier;
- this.requestBufferBuilderFunction = requestBufferBuilderFunction;
- this.requestBufferBuilderBlockingFunction = requestBufferBuilderBlockingFunction;
- this.addBufferListenerFunction = addBufferListenerFunction;
- this.isDestroyedSupplier = isDestroyedSupplier;
- this.requestMemorySegmentSupplier = requestMemorySegmentSupplier;
- this.requestMemorySegmentBlockingSupplier = requestMemorySegmentBlockingSupplier;
- this.getExpectedNumberOfMemorySegmentsSupplier = getExpectedNumberOfMemorySegmentsSupplier;
- this.getMinNumberOfMemorySegmentsSupplier = getMinNumberOfMemorySegmentsSupplier;
- this.getMaxNumberOfMemorySegmentsSupplier = getMaxNumberOfMemorySegmentsSupplier;
- this.getNumBuffersSupplier = getNumBuffersSupplier;
- this.setNumBuffersConsumer = setNumBuffersConsumer;
- this.setMaxOverdraftBuffersPerGateConsumer = setMaxOverdraftBuffersPerGateConsumer;
- this.getMaxOverdraftBuffersPerGateSupplier = getMaxOverdraftBuffersPerGateSupplier;
- this.getNumberOfAvailableMemorySegmentsSupplier =
- getNumberOfAvailableMemorySegmentsSupplier;
- this.bestEffortGetNumOfUsedBuffersSupplier = bestEffortGetNumOfUsedBuffersSupplier;
- this.recycleConsumer = recycleConsumer;
- this.getAvailableFutureSupplier = getAvailableFutureSupplier;
- }
-
- public static TestingBufferPool.Builder builder() {
- return new TestingBufferPool.Builder();
- }
-
@Override
- public void reserveSegments(int numberOfSegmentsToReserve) throws IOException {
- reserveSegmentsConsumer.accept(numberOfSegmentsToReserve);
- }
+ public void reserveSegments(int numberOfSegmentsToReserve) {}
@Override
- public void lazyDestroy() {
- lazyDestroyRunnable.run();
- }
+ public void lazyDestroy() {}
- @Nullable
@Override
public Buffer requestBuffer() {
- return requestBufferSupplier.get();
+ MemorySegment segment = requestMemorySegment();
+ return new NetworkBuffer(segment, FreeingBufferRecycler.INSTANCE);
}
- @Nullable
@Override
public BufferBuilder requestBufferBuilder() {
- return requestBufferBuilderSupplier.get();
+ throw new UnsupportedOperationException();
}
- @Nullable
@Override
- public BufferBuilder requestBufferBuilder(int targetChannel) {
- return requestBufferBuilderFunction.apply(targetChannel);
+ public BufferBuilder requestBufferBuilderBlocking() throws InterruptedException {
+ throw new UnsupportedOperationException();
}
@Override
- public BufferBuilder requestBufferBuilderBlocking() throws InterruptedException {
- return requestBufferBuilderBlockingSupplier.get();
+ public BufferBuilder requestBufferBuilder(int targetChannel) {
+ throw new UnsupportedOperationException();
}
@Override
public BufferBuilder requestBufferBuilderBlocking(int targetChannel)
throws InterruptedException {
- return requestBufferBuilderBlockingFunction.apply(targetChannel);
+ throw new UnsupportedOperationException();
}
@Override
public boolean addBufferListener(BufferListener listener) {
- return addBufferListenerFunction.apply(listener);
+ throw new UnsupportedOperationException();
}
@Override
public boolean isDestroyed() {
- return isDestroyedSupplier.get();
+ throw new UnsupportedOperationException();
}
- @Nullable
@Override
public MemorySegment requestMemorySegment() {
- return requestMemorySegmentSupplier.get();
+ return MemorySegmentFactory.allocateUnpooledSegment(1024);
}
@Override
public MemorySegment requestMemorySegmentBlocking() throws InterruptedException {
- return requestMemorySegmentBlockingSupplier.get();
+ throw new UnsupportedOperationException();
}
@Override
public int getExpectedNumberOfMemorySegments() {
- return getExpectedNumberOfMemorySegmentsSupplier.get();
+ throw new UnsupportedOperationException();
}
@Override
public int getMinNumberOfMemorySegments() {
- return getMinNumberOfMemorySegmentsSupplier.get();
+ throw new UnsupportedOperationException();
}
@Override
public int getMaxNumberOfMemorySegments() {
- return getMaxNumberOfMemorySegmentsSupplier.get();
+ throw new UnsupportedOperationException();
}
@Override
public int getNumBuffers() {
- return getNumBuffersSupplier.get();
+ return 0;
}
@Override
public void setNumBuffers(int numBuffers) {
- setNumBuffersConsumer.accept(numBuffers);
+ throw new UnsupportedOperationException();
}
@Override
public void setMaxOverdraftBuffersPerGate(int maxOverdraftBuffersPerGate) {
- setMaxOverdraftBuffersPerGateConsumer.accept(maxOverdraftBuffersPerGate);
+ throw new UnsupportedOperationException();
}
@Override
public int getMaxOverdraftBuffersPerGate() {
- return getMaxOverdraftBuffersPerGateSupplier.get();
+ throw new UnsupportedOperationException();
}
@Override
public int getNumberOfAvailableMemorySegments() {
- return getNumberOfAvailableMemorySegmentsSupplier.get();
+ throw new UnsupportedOperationException();
}
@Override
public int bestEffortGetNumOfUsedBuffers() {
- return bestEffortGetNumOfUsedBuffersSupplier.get();
+ throw new UnsupportedOperationException();
}
@Override
public void recycle(MemorySegment memorySegment) {
- recycleConsumer.accept(memorySegment);
+ throw new UnsupportedOperationException();
}
@Override
public CompletableFuture<?> getAvailableFuture() {
- return getAvailableFutureSupplier.get();
- }
-
- /** Builder for {@link TestingBufferPool}. */
- public static class Builder {
-
- private Consumer<Integer> reserveSegmentsConsumer = (ignore) -> {};
-
- private Runnable lazyDestroyRunnable = () -> {};
-
- private Supplier<Buffer> requestBufferSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<BufferBuilder> requestBufferBuilderSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<BufferBuilder> requestBufferBuilderBlockingSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Function<Integer, BufferBuilder> requestBufferBuilderFunction =
- (ignore) -> {
- throw new UnsupportedOperationException();
- };
-
- private Function<Integer, BufferBuilder> requestBufferBuilderBlockingFunction =
- (ignore) -> {
- throw new UnsupportedOperationException();
- };
-
- private Function<BufferListener, Boolean> addBufferListenerFunction =
- (ignore) -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Boolean> isDestroyedSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<MemorySegment> requestMemorySegmentSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<MemorySegment> requestMemorySegmentBlockingSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Integer> getExpectedNumberOfMemorySegmentsSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Integer> getMinNumberOfMemorySegmentsSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Integer> getMaxNumberOfMemorySegmentsSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Integer> getNumBuffersSupplier = () -> 0;
-
- private Consumer<Integer> setNumBuffersConsumer =
- (ignore) -> {
- throw new UnsupportedOperationException();
- };
-
- private Consumer<Integer> setMaxOverdraftBuffersPerGateConsumer =
- (ignore) -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Integer> getMaxOverdraftBuffersPerGateSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Integer> getNumberOfAvailableMemorySegmentsSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<Integer> bestEffortGetNumOfUsedBuffersSupplier =
- () -> {
- throw new UnsupportedOperationException();
- };
-
- private Consumer<MemorySegment> recycleConsumer =
- (ignore) -> {
- throw new UnsupportedOperationException();
- };
-
- private Supplier<CompletableFuture<?>> getAvailableFutureSupplier = () -> AVAILABLE;
-
- private Builder() {}
-
- public TestingBufferPool build() {
- return new TestingBufferPool(
- reserveSegmentsConsumer,
- lazyDestroyRunnable,
- requestBufferSupplier,
- requestBufferBuilderSupplier,
- requestBufferBuilderBlockingSupplier,
- requestBufferBuilderFunction,
- requestBufferBuilderBlockingFunction,
- addBufferListenerFunction,
- isDestroyedSupplier,
- requestMemorySegmentSupplier,
- requestMemorySegmentBlockingSupplier,
- getExpectedNumberOfMemorySegmentsSupplier,
- getMinNumberOfMemorySegmentsSupplier,
- getMaxNumberOfMemorySegmentsSupplier,
- getNumBuffersSupplier,
- setNumBuffersConsumer,
- setMaxOverdraftBuffersPerGateConsumer,
- getMaxOverdraftBuffersPerGateSupplier,
- getNumberOfAvailableMemorySegmentsSupplier,
- bestEffortGetNumOfUsedBuffersSupplier,
- recycleConsumer,
- getAvailableFutureSupplier);
- }
-
- public Builder setReserveSegmentsConsumer(Consumer<Integer> reserveSegmentsConsumer) {
- this.reserveSegmentsConsumer = reserveSegmentsConsumer;
- return this;
- }
-
- public Builder setLazyDestroyRunnable(Runnable lazyDestroyRunnable) {
- this.lazyDestroyRunnable = lazyDestroyRunnable;
- return this;
- }
-
- public Builder setRequestBufferSupplier(Supplier<Buffer> requestBufferSupplier) {
- this.requestBufferSupplier = requestBufferSupplier;
- return this;
- }
-
- public Builder setRequestBufferBuilderSupplier(
- Supplier<BufferBuilder> requestBufferBuilderSupplier) {
- this.requestBufferBuilderSupplier = requestBufferBuilderSupplier;
- return this;
- }
-
- public Builder setRequestBufferBuilderBlockingSupplier(
- Supplier<BufferBuilder> requestBufferBuilderBlockingSupplier) {
- this.requestBufferBuilderBlockingSupplier = requestBufferBuilderBlockingSupplier;
- return this;
- }
-
- public Builder setRequestBufferBuilderFunction(
- Function<Integer, BufferBuilder> requestBufferBuilderFunction) {
- this.requestBufferBuilderFunction = requestBufferBuilderFunction;
- return this;
- }
-
- public Builder setRequestBufferBuilderBlockingFunction(
- Function<Integer, BufferBuilder> requestBufferBuilderBlockingFunction) {
- this.requestBufferBuilderBlockingFunction = requestBufferBuilderBlockingFunction;
- return this;
- }
-
- public Builder setAddBufferListenerFunction(
- Function<BufferListener, Boolean> addBufferListenerFunction) {
- this.addBufferListenerFunction = addBufferListenerFunction;
- return this;
- }
-
- public Builder setIsDestroyedSupplier(Supplier<Boolean> isDestroyedSupplier) {
- this.isDestroyedSupplier = isDestroyedSupplier;
- return this;
- }
-
- public Builder setRequestMemorySegmentSupplier(
- Supplier<MemorySegment> requestMemorySegmentSupplier) {
- this.requestMemorySegmentSupplier = requestMemorySegmentSupplier;
- return this;
- }
-
- public Builder setRequestMemorySegmentBlockingSupplier(
- Supplier<MemorySegment> requestMemorySegmentBlockingSupplier) {
- this.requestMemorySegmentBlockingSupplier = requestMemorySegmentBlockingSupplier;
- return this;
- }
-
- public Builder setGetExpectedNumberOfMemorySegmentsSupplier(
- Supplier<Integer> getExpectedNumberOfMemorySegmentsSupplier) {
- this.getExpectedNumberOfMemorySegmentsSupplier =
- getExpectedNumberOfMemorySegmentsSupplier;
- return this;
- }
-
- public Builder setGetMinNumberOfMemorySegmentsSupplier(
- Supplier<Integer> getMinNumberOfMemorySegmentsSupplier) {
- this.getMinNumberOfMemorySegmentsSupplier = getMinNumberOfMemorySegmentsSupplier;
- return this;
- }
-
- public Builder setGetMaxNumberOfMemorySegmentsSupplier(
- Supplier<Integer> getMaxNumberOfMemorySegmentsSupplier) {
- this.getMaxNumberOfMemorySegmentsSupplier = getMaxNumberOfMemorySegmentsSupplier;
- return this;
- }
-
- public Builder setGetNumBuffersSupplier(Supplier<Integer> getNumBuffersSupplier) {
- this.getNumBuffersSupplier = getNumBuffersSupplier;
- return this;
- }
-
- public Builder setSetNumBuffersConsumer(Consumer<Integer> setNumBuffersConsumer) {
- this.setNumBuffersConsumer = setNumBuffersConsumer;
- return this;
- }
-
- public Builder setSetMaxOverdraftBuffersPerGateConsumer(
- Consumer<Integer> setMaxOverdraftBuffersPerGateConsumer) {
- this.setMaxOverdraftBuffersPerGateConsumer = setMaxOverdraftBuffersPerGateConsumer;
- return this;
- }
-
- public Builder setGetMaxOverdraftBuffersPerGateSupplier(
- Supplier<Integer> getMaxOverdraftBuffersPerGateSupplier) {
- this.getMaxOverdraftBuffersPerGateSupplier = getMaxOverdraftBuffersPerGateSupplier;
- return this;
- }
-
- public Builder setGetNumberOfAvailableMemorySegmentsSupplier(
- Supplier<Integer> getNumberOfAvailableMemorySegmentsSupplier) {
- this.getNumberOfAvailableMemorySegmentsSupplier =
- getNumberOfAvailableMemorySegmentsSupplier;
- return this;
- }
-
- public Builder setBestEffortGetNumOfUsedBuffersSupplier(
- Supplier<Integer> bestEffortGetNumOfUsedBuffersSupplier) {
- this.bestEffortGetNumOfUsedBuffersSupplier = bestEffortGetNumOfUsedBuffersSupplier;
- return this;
- }
-
- public Builder setRecycleConsumer(Consumer<MemorySegment> recycleConsumer) {
- this.recycleConsumer = recycleConsumer;
- return this;
- }
-
- public Builder setGetAvailableFutureSupplier(
- Supplier<CompletableFuture<?>> getAvailableFutureSupplier) {
- this.getAvailableFutureSupplier = getAvailableFutureSupplier;
- return this;
- }
+ return AVAILABLE;
}
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
index 8a6c440def3..7d78fba72bb 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
@@ -19,7 +19,6 @@
package org.apache.flink.runtime.io.network.netty;
import org.apache.flink.configuration.NettyShuffleEnvironmentOptions.CompressionCodec;
-import org.apache.flink.core.memory.MemorySegmentFactory;
import org.apache.flink.metrics.SimpleCounter;
import org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter;
import org.apache.flink.runtime.clusterframework.types.ResourceID;
@@ -596,11 +595,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
final String expectedMessage = "test exception on buffer";
final NetworkBufferPool networkBufferPool = new NetworkBufferPool(10, bufferSize);
final SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
- inputGate.setBufferPool(
- TestingBufferPool.builder()
- .setRequestMemorySegmentSupplier(
- () -> MemorySegmentFactory.allocateUnpooledSegment(1024))
- .build());
+ inputGate.setBufferPool(new TestingBufferPool());
final RemoteInputChannel inputChannel =
new TestRemoteInputChannelForError(inputGate, expectedMessage);
final CreditBasedPartitionRequestClientHandler handler =
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java
index 90c46267800..96611550686 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java
@@ -78,11 +78,7 @@ class NettyMessageClientDecoderDelegateTest {
channel = new EmbeddedChannel(new NettyMessageClientDecoderDelegate(handler));
inputGate = createSingleInputGate(1, networkBufferPool);
- inputGate.setBufferPool(
- TestingBufferPool.builder()
- .setRequestMemorySegmentSupplier(
- () -> MemorySegmentFactory.allocateUnpooledSegment(1024))
- .build());
+ inputGate.setBufferPool(new TestingBufferPool());
RemoteInputChannel inputChannel =
createRemoteInputChannel(
inputGate, new TestingPartitionRequestClient(), NUMBER_OF_BUFFER_RESPONSES);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java
index 39549a5623a..8f4123a08a6 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java
@@ -81,11 +81,7 @@ class NettyMessageClientSideSerializationTest {
void setup() throws IOException, InterruptedException {
networkBufferPool = new NetworkBufferPool(8, BUFFER_SIZE);
inputGate = createSingleInputGate(1, networkBufferPool);
- inputGate.setBufferPool(
- TestingBufferPool.builder()
- .setRequestMemorySegmentSupplier(
- () -> MemorySegmentFactory.allocateUnpooledSegment(1024))
- .build());
+ inputGate.setBufferPool(new TestingBufferPool());
RemoteInputChannel inputChannel =
createRemoteInputChannel(inputGate, new TestingPartitionRequestClient());
inputChannel.requestSubpartitions();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
index 8be355781f9..419bf508d8e 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
@@ -18,7 +18,6 @@
package org.apache.flink.runtime.io.network.partition;
-import org.apache.flink.core.memory.MemorySegmentFactory;
import org.apache.flink.runtime.io.network.ConnectionManager;
import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
@@ -273,11 +272,7 @@ public class InputGateFairnessTest {
channels[11].onBuffer(mockBuffer, 0, -1, 0);
channelSequenceNums[11]++;
- gate.setBufferPool(
- TestingBufferPool.builder()
- .setRequestMemorySegmentSupplier(
- () -> MemorySegmentFactory.allocateUnpooledSegment(1024))
- .build());
+ gate.setBufferPool(new TestingBufferPool());
gate.setInputChannels(channels);
gate.setupChannels();
gate.requestPartitions();
@@ -358,7 +353,7 @@ public class InputGateFairnessTest {
private static class FairnessVerifyingInputGate extends SingleInputGate {
private static final int BUFFER_SIZE = 32 * 1024;
private static final SupplierWithException<BufferPool, IOException>
- STUB_BUFFER_POOL_FACTORY = () -> TestingBufferPool.NO_OP;
+ STUB_BUFFER_POOL_FACTORY = TestingBufferPool::new;
private final PrioritizedDeque<InputChannel> channelsWithData;
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java
index 88e5f64dcdf..b53a23ca93f 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java
@@ -94,8 +94,6 @@ public class ResultPartitionBuilder {
private long hybridShuffleNumRetainedInMemoryRegionsMax = Long.MAX_VALUE;
- private boolean isMemoryDecouplingEnabled = false;
-
public ResultPartitionBuilder setResultPartitionIndex(int partitionIndex) {
this.partitionIndex = partitionIndex;
return this;
@@ -236,11 +234,6 @@ public class ResultPartitionBuilder {
return this;
}
- public ResultPartitionBuilder setIsMemoryDecouplingEnabled(boolean isMemoryDecouplingEnabled) {
- this.isMemoryDecouplingEnabled = isMemoryDecouplingEnabled;
- return this;
- }
-
public ResultPartitionBuilder setHybridShuffleSpilledIndexRegionGroupSize(
int hybridShuffleSpilledIndexRegionGroupSize) {
this.hybridShuffleSpilledIndexRegionGroupSize = hybridShuffleSpilledIndexRegionGroupSize;
@@ -268,7 +261,6 @@ public class ResultPartitionBuilder {
maxOverdraftBuffersPerGate,
hybridShuffleSpilledIndexRegionGroupSize,
hybridShuffleNumRetainedInMemoryRegionsMax,
- isMemoryDecouplingEnabled,
Optional.empty());
SupplierWithException<BufferPool, IOException> factory =
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactoryTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactoryTest.java
index 88abb961672..0ba2e01908d 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactoryTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactoryTest.java
@@ -221,7 +221,6 @@ class ResultPartitionFactoryTest {
0,
256,
Long.MAX_VALUE,
- false,
tieredStorage);
final ResultPartitionDeploymentDescriptor descriptor =
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java
index fbab00c57a2..1652f80fe91 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java
@@ -121,9 +121,9 @@ class GateBuffersSpecTest {
createGateBuffersSpec(
numInputChannels, partitionType, numExclusiveBuffersPerChannel);
- int numExclusivePerChannel = 0;
int minBuffersPerGate = 1;
int maxBuffersPerGate = 8;
+ int numExclusivePerChannel = 0;
checkBuffersInGate(
gateBuffersSpec,
@@ -173,36 +173,6 @@ class GateBuffersSpecTest {
assertThat(effectiveMaxRequiredBuffers).isEqualTo(expectEffectiveMaxRequiredBuffers);
}
- @ParameterizedTest
- @MethodSource("parameters")
- void testDefaultMinBuffersPerGate(ResultPartitionType partitionType) {
- int numInputChannels = 499;
- int numExclusiveBuffers = 2;
-
- GateBuffersSpec gateBuffersSpec =
- createGateBuffersSpec(
- numInputChannels, partitionType, numExclusiveBuffers, false, false);
- int expectedBuffersPerGate = 999;
- int maxBuffersPerGate = 1006;
- checkBuffersInGate(
- gateBuffersSpec,
- numExclusiveBuffers,
- expectedBuffersPerGate,
- expectedBuffersPerGate,
- maxBuffersPerGate);
-
- gateBuffersSpec =
- createGateBuffersSpec(
- numInputChannels, partitionType, numExclusiveBuffers, true, true);
- int minBuffersPerGate = partitionType.isHybridResultPartition() ? 2 : 999;
- checkBuffersInGate(
- gateBuffersSpec,
- numExclusiveBuffers,
- expectedBuffersPerGate,
- minBuffersPerGate,
- maxBuffersPerGate);
- }
-
private static void checkBuffersInGate(
GateBuffersSpec gateBuffersSpec,
int numExclusivePerChannel,
@@ -231,26 +201,9 @@ class GateBuffersSpecTest {
8,
partitionType,
numInputChannels,
- false,
false);
}
- private static GateBuffersSpec createGateBuffersSpec(
- int numInputChannels,
- ResultPartitionType partitionType,
- int numExclusiveBuffersPerChannel,
- boolean enableTieredStorage,
- boolean memoryDecouplingEnabled) {
- return InputGateSpecUtils.createGateBuffersSpec(
- getMaxRequiredBuffersPerGate(partitionType),
- numExclusiveBuffersPerChannel,
- 8,
- partitionType,
- numInputChannels,
- enableTieredStorage,
- memoryDecouplingEnabled);
- }
-
private static Optional<Integer> getMaxRequiredBuffersPerGate(
ResultPartitionType partitionType) {
return isPipelinedOrHybridResultPartition(partitionType)
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
index 0dc60e9e3d6..6abadbf10ea 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
@@ -18,7 +18,6 @@
package org.apache.flink.runtime.io.network.partition.consumer;
-import org.apache.flink.core.memory.MemorySegmentFactory;
import org.apache.flink.core.testutils.OneShotLatch;
import org.apache.flink.runtime.checkpoint.CheckpointException;
import org.apache.flink.runtime.checkpoint.CheckpointOptions;
@@ -40,8 +39,6 @@ import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.buffer.Buffer.DataType;
import org.apache.flink.runtime.io.network.buffer.BufferBuilder;
import org.apache.flink.runtime.io.network.buffer.BufferPool;
-import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
-import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
import org.apache.flink.runtime.io.network.partition.InputChannelTestUtils;
@@ -1378,14 +1375,7 @@ class RemoteInputChannelTest {
@Test
void testOnUpstreamBlockedAndResumed() throws Exception {
- BufferPool bufferPool =
- TestingBufferPool.builder()
- .setRequestBufferSupplier(
- () ->
- new NetworkBuffer(
- MemorySegmentFactory.allocateUnpooledSegment(1024),
- FreeingBufferRecycler.INSTANCE))
- .build();
+ BufferPool bufferPool = new TestingBufferPool();
SingleInputGate inputGate = createSingleInputGate(bufferPool);
RemoteInputChannel remoteChannel1 = createRemoteInputChannel(inputGate, 0, 2);
@@ -1434,14 +1424,7 @@ class RemoteInputChannelTest {
@Test
void testRequestBuffer() throws Exception {
- BufferPool bufferPool =
- TestingBufferPool.builder()
- .setRequestBufferSupplier(
- () ->
- new NetworkBuffer(
- MemorySegmentFactory.allocateUnpooledSegment(1024),
- FreeingBufferRecycler.INSTANCE))
- .build();
+ BufferPool bufferPool = new TestingBufferPool();
SingleInputGate inputGate = createSingleInputGate(bufferPool);
RemoteInputChannel remoteChannel1 = createRemoteInputChannel(inputGate, 0, 2);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
index 99226e0c1d0..ab89c226c29 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
@@ -70,13 +70,15 @@ public class SingleInputGateBuilder {
private MemorySegmentProvider segmentProvider =
InputChannelTestUtils.StubMemorySegmentProvider.getInstance();
+ private BufferPool bufferPool = null;
+
private ChannelStateWriter channelStateWriter = ChannelStateWriter.NO_OP;
@Nullable
private BiFunction<InputChannelBuilder, SingleInputGate, InputChannel> channelFactory = null;
private SupplierWithException<BufferPool, IOException> bufferPoolFactory =
- () -> TestingBufferPool.NO_OP;
+ TestingBufferPool::new;
private BufferDebloatConfiguration bufferDebloatConfiguration =
BufferDebloatConfiguration.fromConfiguration(new Configuration());
private Function<BufferDebloatConfiguration, ThroughputCalculator> createThroughputCalculator =
@@ -147,6 +149,11 @@ public class SingleInputGateBuilder {
return this;
}
+ public SingleInputGateBuilder setBufferPool(BufferPool bufferPool) {
+ this.bufferPool = bufferPool;
+ return this;
+ }
+
/** Adds automatic initialization of all channels with the given factory. */
public SingleInputGateBuilder setChannelFactory(
BiFunction<InputChannelBuilder, SingleInputGate, InputChannel> channelFactory) {
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/TestingTieredStorageMemoryManager.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/TestingTieredStorageMemoryManager.java
index 649346ca8e5..a4b2349fd34 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/TestingTieredStorageMemoryManager.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/TestingTieredStorageMemoryManager.java
@@ -53,8 +53,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
private final TriConsumer<Object, Object, Buffer> transferBufferOwnershipConsumer;
- private final Supplier<Integer> getBufferPoolSizeSupplier;
-
private final Runnable releaseRunnable;
private TestingTieredStorageMemoryManager(
@@ -67,7 +65,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
Function<Integer, Boolean> ensureCapacityFunction,
Function<Object, Integer> numOwnerRequestedBufferFunction,
TriConsumer<Object, Object, Buffer> transferBufferOwnershipConsumer,
- Supplier<Integer> getBufferPoolSizeSupplier,
Runnable releaseRunnable) {
this.setupConsumer = setupConsumer;
this.setMetricGroupConsumer = setMetricGroupConsumer;
@@ -78,7 +75,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
this.ensureCapacityFunction = ensureCapacityFunction;
this.numOwnerRequestedBufferFunction = numOwnerRequestedBufferFunction;
this.transferBufferOwnershipConsumer = transferBufferOwnershipConsumer;
- this.getBufferPoolSizeSupplier = getBufferPoolSizeSupplier;
this.releaseRunnable = releaseRunnable;
}
@@ -127,11 +123,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
transferBufferOwnershipConsumer.accept(oldOwner, newOwner, buffer);
}
- @Override
- public int getBufferPoolSize() {
- return getBufferPoolSizeSupplier.get();
- }
-
@Override
public void release() {
releaseRunnable.run();
@@ -149,8 +140,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
private Consumer<Runnable> listenBufferReclaimRequestConsumer = runnable -> {};
- private Function<Object, BufferBuilder> requestBufferFunction = owner -> null;
-
private Function<Object, BufferBuilder> requestBufferBlockingFunction = owner -> null;
private Function<Object, Integer> getMaxNonReclaimableBuffersFunction = owner -> 0;
@@ -162,8 +151,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
private TriConsumer<Object, Object, Buffer> transferBufferOwnershipConsumer =
(oldOwner, newOwner, buffer) -> {};
- private Supplier<Integer> getBufferPoolSizeSupplier = () -> 0;
-
private Runnable releaseRunnable = () -> {};
public Builder() {}
@@ -216,12 +203,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
return this;
}
- public TestingTieredStorageMemoryManager.Builder setGetBufferPoolSizeSupplier(
- Supplier<Integer> getBufferPoolSizeSupplier) {
- this.getBufferPoolSizeSupplier = getBufferPoolSizeSupplier;
- return this;
- }
-
public TestingTieredStorageMemoryManager.Builder setReleaseRunnable(
Runnable releaseRunnable) {
this.releaseRunnable = releaseRunnable;
@@ -239,7 +220,6 @@ public class TestingTieredStorageMemoryManager implements TieredStorageMemoryMan
ensureCapacityFunction,
numOwnerRequestedBufferFunction,
transferBufferOwnershipConsumer,
- getBufferPoolSizeSupplier,
releaseRunnable);
}
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java
index e2d771a5c72..865464fc1e0 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java
@@ -147,9 +147,8 @@ class TieredResultPartitionTest {
@Test
void testMinMaxNetworkBuffersTieredResultPartition() {
- final int numSubpartitions = 105;
- final int tieredStorageTotalExclusiveBufferNum = 103;
- final int tieredStorageMaxBuffersPerRecord = 8;
+ int numSubpartitions = 105;
+ int tieredStorageTotalExclusiveBufferNum = 103;
Tuple3<Integer, Integer, Integer> minMaxNetworkBuffers =
getMinMaxNetworkBuffersPerResultPartition(
100,
@@ -158,12 +157,10 @@ class TieredResultPartitionTest {
10,
numSubpartitions,
true,
- true,
tieredStorageTotalExclusiveBufferNum,
- tieredStorageMaxBuffersPerRecord,
ResultPartitionType.HYBRID_SELECTIVE);
assertThat(minMaxNetworkBuffers.f0).isEqualTo(tieredStorageTotalExclusiveBufferNum);
- assertThat(minMaxNetworkBuffers.f1).isEqualTo(tieredStorageMaxBuffersPerRecord);
+ assertThat(minMaxNetworkBuffers.f1).isEqualTo(8);
assertThat(minMaxNetworkBuffers.f2).isEqualTo(Integer.MAX_VALUE);
}
@@ -296,7 +293,6 @@ class TieredResultPartitionTest {
Integer.MAX_VALUE,
NETWORK_BUFFER_SIZE,
isBroadcastOnly,
- true,
new ResultPartitionManager(),
new BufferCompressor(NETWORK_BUFFER_SIZE, CompressionCodec.LZ4),
tierShuffleDescriptors,
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java
index 62ebaeb5f98..798ee7d7bc4 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java
@@ -95,7 +95,7 @@ class SortBufferAccumulatorTest {
// flushed.
try (SortBufferAccumulator bufferAccumulator =
new SortBufferAccumulator(
- 1, 2, BUFFER_SIZE_BYTES, 0L, memoryManager, isPartialRecordAllowed)) {
+ 1, 2, BUFFER_SIZE_BYTES, memoryManager, isPartialRecordAllowed)) {
bufferAccumulator.setup(
((subpartition, buffer, numRemainingBuffers) -> {
assertThat(buffer.getDataType()).isIn(expectedDataTypes);
@@ -144,7 +144,7 @@ class SortBufferAccumulatorTest {
// flushed.
try (SortBufferAccumulator bufferAccumulator =
new SortBufferAccumulator(
- 1, 2, BUFFER_SIZE_BYTES, 0L, memoryManager, isPartialRecordAllowed)) {
+ 1, 2, BUFFER_SIZE_BYTES, memoryManager, isPartialRecordAllowed)) {
AtomicInteger numReceivedBuffers = new AtomicInteger(0);
bufferAccumulator.setup(
(subpartitionIndex, buffer, numRemainingBuffers) -> {
@@ -170,8 +170,8 @@ class SortBufferAccumulatorTest {
TieredStorageMemoryManager memoryManager = createStorageMemoryManager(numBuffers);
try (SortBufferAccumulator bufferAccumulator =
- new SortBufferAccumulator(1, 1, bufferSize, 0L, memoryManager, true)) {
- bufferAccumulator.setup((subpartitionIndex, buffers, numRemainingBuffers) -> {});
+ new SortBufferAccumulator(1, 1, bufferSize, memoryManager, true)) {
+ bufferAccumulator.setup((subpartitionIndex, buffer, numRemainingBuffers) -> {});
assertThatThrownBy(
() ->
bufferAccumulator.receive(
@@ -191,7 +191,7 @@ class SortBufferAccumulatorTest {
createStorageMemoryManager(numBuffers);
SortBufferAccumulator bufferAccumulator =
new SortBufferAccumulator(
- 1, 2, BUFFER_SIZE_BYTES, 0L, tieredStorageMemoryManager, true);
+ 1, 2, BUFFER_SIZE_BYTES, tieredStorageMemoryManager, true);
bufferAccumulator.setup(
((subpartition, buffer, numRemainingBuffers) -> buffer.recycleBuffer()));
bufferAccumulator.receive(
@@ -221,17 +221,16 @@ class SortBufferAccumulatorTest {
// When flushing data from the data buffer, the buffers should
// not be requested from here anymore because the freeSegments
// can be reused before released.
- assertThat(numRequestedBuffers.get() < numBuffers).isTrue();
+ assertThat(numRequestedBuffers.get() <= numBuffers).isTrue();
return new BufferBuilder(
memorySegment,
segment -> globalPool.requestPooledMemorySegment());
})
- .setGetBufferPoolSizeSupplier(() -> numBuffers)
.build();
SortBufferAccumulator bufferAccumulator =
new SortBufferAccumulator(
- numSubpartitions, numBuffers, BUFFER_SIZE_BYTES, 0L, memoryManager, true);
+ numSubpartitions, numBuffers, BUFFER_SIZE_BYTES, memoryManager, true);
bufferAccumulator.setup(
((subpartition, buffer, numRemainingBuffers) -> buffer.recycleBuffer()));
diff --git a/flink-tests/src/test/java/org/apache/flink/test/runtime/BatchShuffleITCaseBase.java b/flink-tests/src/test/java/org/apache/flink/test/runtime/BatchShuffleITCaseBase.java
index 6c693d5e373..3f03c06d287 100644
--- a/flink-tests/src/test/java/org/apache/flink/test/runtime/BatchShuffleITCaseBase.java
+++ b/flink-tests/src/test/java/org/apache/flink/test/runtime/BatchShuffleITCaseBase.java
@@ -136,16 +136,15 @@ class BatchShuffleITCaseBase {
throws Exception {
JobGraphRunningUtil.execute(
jobGraph, configuration, NUM_TASK_MANAGERS, NUM_SLOTS_PER_TASK_MANAGER);
- checkAllDataReceived(numRecordsToSend, PARALLELISM, 1);
+ checkAllDataReceived(numRecordsToSend);
}
- protected void checkAllDataReceived(
- int numRecordsToSend, int parallelism, int numResultPartitions) {
+ private void checkAllDataReceived(int numRecordsToSend) {
assertThat(Arrays.stream(NUM_RECEIVED_RECORDS).sum())
- .isEqualTo(numRecordsToSend * parallelism * numResultPartitions);
+ .isEqualTo(numRecordsToSend * PARALLELISM);
}
- static class StringSource implements ParallelSourceFunction<String> {
+ private static class StringSource implements ParallelSourceFunction<String> {
private volatile boolean isRunning = true;
private int numRecordsToSend;
@@ -166,7 +165,7 @@ class BatchShuffleITCaseBase {
}
}
- static class VerifySink extends RichSinkFunction<String> {
+ private static class VerifySink extends RichSinkFunction<String> {
private final boolean failExecution;
private final boolean deletePartitionFile;
@@ -181,14 +180,9 @@ class BatchShuffleITCaseBase {
@Override
public void open(OpenContext openContext) throws Exception {
- int indexOfSubtask = getRuntimeContext().getTaskInfo().getIndexOfThisSubtask();
- int attemptNumber = getRuntimeContext().getTaskInfo().getAttemptNumber();
-
- if (attemptNumber > 0) {
- NUM_RECEIVED_RECORDS[indexOfSubtask] = 0;
- }
-
- if (attemptNumber > 0 || indexOfSubtask != 0) {
+ NUM_RECEIVED_RECORDS[getRuntimeContext().getTaskInfo().getIndexOfThisSubtask()] = 0;
+ if (getRuntimeContext().getTaskInfo().getAttemptNumber() > 0
+ || getRuntimeContext().getTaskInfo().getIndexOfThisSubtask() != 0) {
return;
}
diff --git a/flink-tests/src/test/java/org/apache/flink/test/runtime/HybridShuffleMemoryDecouplingITCase.java b/flink-tests/src/test/java/org/apache/flink/test/runtime/HybridShuffleMemoryDecouplingITCase.java
deleted file mode 100644
index 89b00590334..00000000000
--- a/flink-tests/src/test/java/org/apache/flink/test/runtime/HybridShuffleMemoryDecouplingITCase.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.test.runtime;
-
-import org.apache.flink.api.common.BatchShuffleMode;
-import org.apache.flink.api.common.restartstrategy.RestartStrategies;
-import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
-import org.apache.flink.api.connector.source.Boundedness;
-import org.apache.flink.configuration.BatchExecutionOptions;
-import org.apache.flink.configuration.Configuration;
-import org.apache.flink.configuration.ExecutionOptions;
-import org.apache.flink.configuration.NettyShuffleEnvironmentOptions;
-import org.apache.flink.configuration.TaskManagerOptions;
-import org.apache.flink.runtime.jobgraph.JobGraph;
-import org.apache.flink.runtime.jobgraph.JobType;
-import org.apache.flink.streaming.api.datastream.DataStream;
-import org.apache.flink.streaming.api.datastream.DataStreamSource;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.apache.flink.streaming.api.graph.StreamGraph;
-import org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator;
-import org.apache.flink.streaming.api.operators.StreamSource;
-
-import org.junit.jupiter.api.Test;
-
-import static org.assertj.core.api.Assertions.assertThatThrownBy;
-
-/** Tests for hybrid shuffle with memory decoupling enabled. */
-public class HybridShuffleMemoryDecouplingITCase extends BatchShuffleITCaseBase {
-
- private static final int NUM_RECORDS = 10000;
- private static final int NUM_RESULT_PARTITIONS = 10;
- private static final int PARALLELISM = 10;
-
- @Test
- void testHybridWithMultipleResultPartitionsCauseInsufficientBuffers() {
- Configuration configuration = configureHybridOptions(getConfiguration(), false);
- JobGraph jobGraph =
- createJobGraphWithMultipleResultPartitions(
- NUM_RECORDS, false, NUM_RESULT_PARTITIONS, PARALLELISM, configuration);
-
- assertThatThrownBy(() -> JobGraphRunningUtil.execute(jobGraph, configuration, 1, 1))
- .hasStackTraceContaining("Insufficient number of network buffers");
- }
-
- @Test
- void testHybridWithMultipleResultPartitionsWithMemoryDecoupling() throws Exception {
- Configuration configuration = configureHybridOptions(getConfiguration(), true);
- JobGraph jobGraph =
- createJobGraphWithMultipleResultPartitions(
- NUM_RECORDS, false, NUM_RESULT_PARTITIONS, PARALLELISM, configuration);
-
- JobGraphRunningUtil.execute(jobGraph, configuration, 1, 1);
- checkAllDataReceived(NUM_RECORDS, PARALLELISM, NUM_RESULT_PARTITIONS);
- }
-
- private JobGraph createJobGraphWithMultipleResultPartitions(
- int numRecordsToSend,
- boolean failExecution,
- int numResultPartitions,
- int parallelism,
- Configuration configuration) {
- configuration.setBoolean(BatchExecutionOptions.ADAPTIVE_AUTO_PARALLELISM_ENABLED, false);
- StreamExecutionEnvironment env =
- StreamExecutionEnvironment.getExecutionEnvironment(configuration);
- env.setRestartStrategy(RestartStrategies.fixedDelayRestart(10, 0L));
- env.setParallelism(parallelism);
-
- DataStream<String> source =
- new DataStreamSource<>(
- env,
- BasicTypeInfo.STRING_TYPE_INFO,
- new StreamSource<>(new StringSource(numRecordsToSend)),
- true,
- "source",
- Boundedness.BOUNDED);
-
- for (int i = 0; i < numResultPartitions; i++) {
- source.map(value -> value).rebalance().addSink(new VerifySink(failExecution, false));
- }
-
- StreamGraph streamGraph = env.getStreamGraph();
- streamGraph.setJobType(JobType.BATCH);
- return StreamingJobGraphGenerator.createJobGraph(streamGraph);
- }
-
- private Configuration configureHybridOptions(
- Configuration configuration, boolean enableMemoryDecoupling) {
- configuration.set(
- ExecutionOptions.BATCH_SHUFFLE_MODE,
- BatchShuffleMode.ALL_EXCHANGES_HYBRID_SELECTIVE);
- configuration.setBoolean(
- NettyShuffleEnvironmentOptions.NETWORK_HYBRID_SHUFFLE_ENABLE_MEMORY_DECOUPLING,
- enableMemoryDecoupling);
- configuration.setString(TaskManagerOptions.NETWORK_MEMORY_MAX.key(), "3200k");
-
- return configuration;
- }
-}
(flink) 03/06: Revert "[FLINK-33668][runtime] Make the SortBuffer be able to return unused memory segments"
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
guoweijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit af1c8605d4cc294fb01ad13f13d662417b8a0893
Author: Yuxin Tan <ta...@gmail.com>
AuthorDate: Fri Jun 21 13:47:14 2024 +0800
Revert "[FLINK-33668][runtime] Make the SortBuffer be able to return unused memory segments"
This reverts commit ac2ec3496400b594ffe5c711bc1b6343999a8479.
---
.../runtime/io/network/partition/SortBuffer.java | 23 +-----------------
.../tiered/storage/SortBufferAccumulator.java | 16 +------------
.../storage/TieredStorageSortBufferTest.java | 28 ----------------------
3 files changed, 2 insertions(+), 65 deletions(-)
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java
index d10383cc0d6..371484696a6 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java
@@ -76,7 +76,7 @@ public abstract class SortBuffer implements DataBuffer {
private final int bufferSize;
/** Number of guaranteed buffers can be allocated from the buffer pool for data sort. */
- private int numGuaranteedBuffers;
+ private final int numGuaranteedBuffers;
// ---------------------------------------------------------------------------------------------
// Statistics and states
@@ -184,27 +184,6 @@ public abstract class SortBuffer implements DataBuffer {
return false;
}
- /**
- * Try to release some unused memory segments.
- *
- * <p>Note that this class is not thread safe, so please make sure to call {@link
- * #append(ByteBuffer source, int targetSubpartition, Buffer.DataType dataType)} and this method
- * with lock acquired.
- *
- * @param numFreeSegments the number of segments to be released.
- * @return true if released successfully, otherwise false.
- */
- public boolean returnFreeSegments(int numFreeSegments) {
- if (numFreeSegments < numGuaranteedBuffers - segments.size()) {
- for (int i = 0; i < numFreeSegments; i++) {
- bufferRecycler.recycle(freeSegments.poll());
- }
- numGuaranteedBuffers -= numFreeSegments;
- return true;
- }
- return false;
- }
-
private void writeIndex(int subpartitionIndex, int numRecordBytes, Buffer.DataType dataType) {
MemorySegment segment = segments.get(writeSegmentIndex);
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java
index 2f5c469e33a..8dc6eb2d69f 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulator.java
@@ -151,9 +151,7 @@ public class SortBufferAccumulator implements BufferAccumulator {
int newSize = this.memoryManager.getBufferPoolSize();
int oldSize = poolSize.getAndSet(newSize);
if (oldSize > newSize) {
- if (!returnFreeSegments(oldSize - newSize)) {
- flushCurrentDataBuffer();
- }
+ flushCurrentDataBuffer();
}
},
poolSizeCheckInterval,
@@ -298,18 +296,6 @@ public class SortBufferAccumulator implements BufferAccumulator {
}
}
- private boolean returnFreeSegments(int numSegments) {
- synchronized (lock) {
- if (currentDataBuffer == null
- || currentDataBuffer.isReleased()
- || !currentDataBuffer.hasRemaining()) {
- return false;
- } else {
- return currentDataBuffer.returnFreeSegments(numSegments);
- }
- }
- }
-
private void writeLargeRecord(ByteBuffer record, int subpartitionId, Buffer.DataType dataType) {
checkState(dataType != Buffer.DataType.EVENT_BUFFER);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java
index 2e14969812d..b0ab84f86b0 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java
@@ -29,7 +29,6 @@ import org.apache.flink.runtime.io.network.partition.SortBuffer;
import org.junit.jupiter.api.Test;
-import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.ArrayDeque;
import java.util.ArrayList;
@@ -195,33 +194,6 @@ class TieredStorageSortBufferTest {
assertThat(bufferPool.bestEffortGetNumOfUsedBuffers()).isEqualTo(numBuffersForSort);
}
- @Test
- void testReturnFreeSegments() throws IOException, InterruptedException {
- final int bufferPoolSize = 512;
- final int numBuffersForSort = 20;
- NetworkBufferPool globalPool = new NetworkBufferPool(bufferPoolSize, BUFFER_SIZE_BYTES);
- BufferPool bufferPool =
- globalPool.createBufferPool(bufferPoolSize, bufferPoolSize, bufferPoolSize);
-
- LinkedList<MemorySegment> segments = new LinkedList<>();
- for (int i = 0; i < numBuffersForSort; ++i) {
- segments.add(bufferPool.requestMemorySegmentBlocking());
- }
- TieredStorageSortBuffer sortBuffer =
- new TieredStorageSortBuffer(
- segments, bufferPool, 1, BUFFER_SIZE_BYTES, numBuffersForSort, true);
-
- for (int i = 0; i < 5; i++) {
- byte[] bytes = new byte[BUFFER_SIZE_BYTES];
- ByteBuffer record = ByteBuffer.wrap(bytes);
- sortBuffer.append(record, 0, Buffer.DataType.DATA_BUFFER);
- }
- assertThat(sortBuffer.returnFreeSegments(10)).isTrue();
- assertThat(sortBuffer.returnFreeSegments(10)).isFalse();
-
- sortBuffer.finish();
- }
-
private static BufferWithSubpartition copyIntoSegment(SortBuffer dataBuffer) {
MemorySegment segment = MemorySegmentFactory.allocateUnpooledSegment(BUFFER_SIZE_BYTES);
return dataBuffer.getNextBuffer(segment);
(flink) 05/06: Revert "[FLINK-33668][runtime] Decouple the network memory and job topology on input gate side"
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
guoweijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit eaef742964aa3adb04936b4803acf97f9bafeb21
Author: Yuxin Tan <ta...@gmail.com>
AuthorDate: Fri Jun 21 14:14:17 2024 +0800
Revert "[FLINK-33668][runtime] Decouple the network memory and job topology on input gate side"
This reverts commit 2e257896661ad5e0dca521fa274cad3a0bae7a61.
---
.../metrics/CreditBasedInputBuffersUsageGauge.java | 37 ++---
.../metrics/ExclusiveBuffersUsageGauge.java | 2 +-
.../network/metrics/FloatingBuffersUsageGauge.java | 40 ++---
.../network/metrics/NettyShuffleMetricFactory.java | 10 +-
.../network/netty/NettyPartitionRequestClient.java | 2 +-
.../network/partition/consumer/BufferManager.java | 162 +++------------------
.../partition/consumer/InputGateSpecUtils.java | 10 +-
.../consumer/LocalRecoveredInputChannel.java | 15 --
.../partition/consumer/RecoveredInputChannel.java | 10 +-
.../partition/consumer/RemoteInputChannel.java | 30 ++--
.../consumer/RemoteRecoveredInputChannel.java | 1 -
.../partition/consumer/SingleInputGate.java | 13 --
.../partition/consumer/SingleInputGateFactory.java | 31 +---
.../InputChannelRecoveredStateHandlerTest.java | 6 +-
.../SequentialChannelStateReaderImplTest.java | 2 +-
.../io/network/NettyShuffleEnvironmentTest.java | 19 +--
...{TestingBufferPool.java => NoOpBufferPool.java} | 12 +-
...editBasedPartitionRequestClientHandlerTest.java | 21 +--
.../NettyMessageClientDecoderDelegateTest.java | 4 +-
.../NettyMessageClientSideSerializationTest.java | 4 +-
.../network/partition/InputGateFairnessTest.java | 10 +-
.../io/network/partition/PartitionTestUtils.java | 16 --
.../PipelinedSubpartitionWithReadViewTest.java | 3 -
.../network/partition/ResultPartitionBuilder.java | 2 +-
.../io/network/partition/ResultPartitionTest.java | 4 +-
.../consumer/InputBuffersMetricsTest.java | 36 ++---
.../partition/consumer/RemoteInputChannelTest.java | 74 ++++++----
.../partition/consumer/SingleInputGateBuilder.java | 28 +---
.../partition/consumer/SingleInputGateTest.java | 38 ++---
.../SuccessAfterNetworkBuffersFailureITCase.java | 6 +-
30 files changed, 205 insertions(+), 443 deletions(-)
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/CreditBasedInputBuffersUsageGauge.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/CreditBasedInputBuffersUsageGauge.java
index 40c65862997..34613931715 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/CreditBasedInputBuffersUsageGauge.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/CreditBasedInputBuffersUsageGauge.java
@@ -18,9 +18,6 @@
package org.apache.flink.runtime.io.network.metrics;
-import org.apache.flink.runtime.io.network.buffer.BufferPool;
-import org.apache.flink.runtime.io.network.partition.consumer.InputChannel;
-import org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel;
import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate;
import static org.apache.flink.util.Preconditions.checkNotNull;
@@ -31,35 +28,27 @@ import static org.apache.flink.util.Preconditions.checkNotNull;
*/
public class CreditBasedInputBuffersUsageGauge extends AbstractBuffersUsageGauge {
- public CreditBasedInputBuffersUsageGauge(SingleInputGate[] inputGates) {
+ private final FloatingBuffersUsageGauge floatingBuffersUsageGauge;
+ private final ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge;
+
+ public CreditBasedInputBuffersUsageGauge(
+ FloatingBuffersUsageGauge floatingBuffersUsageGauge,
+ ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge,
+ SingleInputGate[] inputGates) {
super(checkNotNull(inputGates));
+ this.floatingBuffersUsageGauge = checkNotNull(floatingBuffersUsageGauge);
+ this.exclusiveBuffersUsageGauge = checkNotNull(exclusiveBuffersUsageGauge);
}
@Override
public int calculateUsedBuffers(SingleInputGate inputGate) {
- BufferPool bufferPool = inputGate.getBufferPool();
- if (bufferPool != null) {
- int numBuffers = bufferPool.bestEffortGetNumOfUsedBuffers();
- for (InputChannel ic : inputGate.inputChannels()) {
- if (ic instanceof RemoteInputChannel) {
- RemoteInputChannel remoteInputChannel = (RemoteInputChannel) ic;
- numBuffers -= remoteInputChannel.unsynchronizedGetFloatingBuffersAvailable();
- numBuffers -=
- remoteInputChannel.getNumExclusiveBuffers()
- - remoteInputChannel.unsynchronizedGetExclusiveBuffersUsed();
- }
- }
- return Math.max(0, numBuffers);
- }
- return 0;
+ return floatingBuffersUsageGauge.calculateUsedBuffers(inputGate)
+ + exclusiveBuffersUsageGauge.calculateUsedBuffers(inputGate);
}
@Override
public int calculateTotalBuffers(SingleInputGate inputGate) {
- BufferPool bufferPool = inputGate.getBufferPool();
- if (bufferPool != null) {
- return inputGate.getBufferPool().getNumBuffers();
- }
- return 0;
+ return floatingBuffersUsageGauge.calculateTotalBuffers(inputGate)
+ + exclusiveBuffersUsageGauge.calculateTotalBuffers(inputGate);
}
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/ExclusiveBuffersUsageGauge.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/ExclusiveBuffersUsageGauge.java
index 67539667916..9e1751639f1 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/ExclusiveBuffersUsageGauge.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/ExclusiveBuffersUsageGauge.java
@@ -47,7 +47,7 @@ public class ExclusiveBuffersUsageGauge extends AbstractBuffersUsageGauge {
int totalExclusiveBuffers = 0;
for (InputChannel ic : inputGate.inputChannels()) {
if (ic instanceof RemoteInputChannel) {
- totalExclusiveBuffers += ((RemoteInputChannel) ic).getNumExclusiveBuffers();
+ totalExclusiveBuffers += ((RemoteInputChannel) ic).getInitialCredit();
}
}
return totalExclusiveBuffers;
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/FloatingBuffersUsageGauge.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/FloatingBuffersUsageGauge.java
index a2abd8c3c2c..1d17a43e99c 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/FloatingBuffersUsageGauge.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/FloatingBuffersUsageGauge.java
@@ -18,6 +18,9 @@
package org.apache.flink.runtime.io.network.metrics;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.partition.consumer.InputChannel;
+import org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel;
import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate;
import static org.apache.flink.util.Preconditions.checkNotNull;
@@ -25,32 +28,33 @@ import static org.apache.flink.util.Preconditions.checkNotNull;
/** Gauge metric measuring the floating buffers usage gauge for {@link SingleInputGate}s. */
public class FloatingBuffersUsageGauge extends AbstractBuffersUsageGauge {
- private final CreditBasedInputBuffersUsageGauge totalBuffersUsageGauge;
- private final ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge;
-
- public FloatingBuffersUsageGauge(
- SingleInputGate[] inputGates,
- CreditBasedInputBuffersUsageGauge totalBuffersUsageGauge,
- ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge) {
+ public FloatingBuffersUsageGauge(SingleInputGate[] inputGates) {
super(checkNotNull(inputGates));
-
- this.totalBuffersUsageGauge = checkNotNull(totalBuffersUsageGauge);
- this.exclusiveBuffersUsageGauge = checkNotNull(exclusiveBuffersUsageGauge);
}
@Override
public int calculateUsedBuffers(SingleInputGate inputGate) {
- return Math.max(
- 0,
- totalBuffersUsageGauge.calculateUsedBuffers(inputGate)
- - exclusiveBuffersUsageGauge.calculateUsedBuffers(inputGate));
+ int availableFloatingBuffers = 0;
+ BufferPool bufferPool = inputGate.getBufferPool();
+ if (bufferPool != null) {
+ int requestedFloatingBuffers = bufferPool.bestEffortGetNumOfUsedBuffers();
+ for (InputChannel ic : inputGate.inputChannels()) {
+ if (ic instanceof RemoteInputChannel) {
+ availableFloatingBuffers +=
+ ((RemoteInputChannel) ic).unsynchronizedGetFloatingBuffersAvailable();
+ }
+ }
+ return Math.max(0, requestedFloatingBuffers - availableFloatingBuffers);
+ }
+ return 0;
}
@Override
public int calculateTotalBuffers(SingleInputGate inputGate) {
- return Math.max(
- 0,
- totalBuffersUsageGauge.calculateTotalBuffers(inputGate)
- - exclusiveBuffersUsageGauge.calculateTotalBuffers(inputGate));
+ BufferPool bufferPool = inputGate.getBufferPool();
+ if (bufferPool != null) {
+ return inputGate.getBufferPool().getNumBuffers();
+ }
+ return 0;
}
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/NettyShuffleMetricFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/NettyShuffleMetricFactory.java
index 59377e4304f..817190732b4 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/NettyShuffleMetricFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/metrics/NettyShuffleMetricFactory.java
@@ -213,13 +213,13 @@ public class NettyShuffleMetricFactory {
buffersGroup.gauge(METRIC_INPUT_QUEUE_LENGTH, new InputBuffersGauge(inputGates));
buffersGroup.gauge(METRIC_INPUT_QUEUE_SIZE, new InputBuffersSizeGauge(inputGates));
- CreditBasedInputBuffersUsageGauge creditBasedInputBuffersUsageGauge =
- new CreditBasedInputBuffersUsageGauge(inputGates);
+ FloatingBuffersUsageGauge floatingBuffersUsageGauge =
+ new FloatingBuffersUsageGauge(inputGates);
ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge =
new ExclusiveBuffersUsageGauge(inputGates);
- FloatingBuffersUsageGauge floatingBuffersUsageGauge =
- new FloatingBuffersUsageGauge(
- inputGates, creditBasedInputBuffersUsageGauge, exclusiveBuffersUsageGauge);
+ CreditBasedInputBuffersUsageGauge creditBasedInputBuffersUsageGauge =
+ new CreditBasedInputBuffersUsageGauge(
+ floatingBuffersUsageGauge, exclusiveBuffersUsageGauge, inputGates);
buffersGroup.gauge(METRIC_INPUT_EXCLUSIVE_BUFFERS_USAGE, exclusiveBuffersUsageGauge);
buffersGroup.gauge(METRIC_INPUT_FLOATING_BUFFERS_USAGE, floatingBuffersUsageGauge);
buffersGroup.gauge(METRIC_INPUT_POOL_USAGE, creditBasedInputBuffersUsageGauge);
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClient.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClient.java
index 9d50f516aa6..7cc52a234e0 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClient.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClient.java
@@ -129,7 +129,7 @@ public class NettyPartitionRequestClient implements PartitionRequestClient {
partitionId,
subpartitionIndexSet,
inputChannel.getInputChannelId(),
- inputChannel.getNumExclusiveBuffers());
+ inputChannel.getInitialCredit());
final ChannelFutureListener listener =
future -> {
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java
index 9a39e82168a..db38025def9 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/BufferManager.java
@@ -35,6 +35,7 @@ import javax.annotation.concurrent.GuardedBy;
import java.io.IOException;
import java.util.ArrayDeque;
import java.util.ArrayList;
+import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Queue;
@@ -47,18 +48,14 @@ import static org.apache.flink.util.Preconditions.checkState;
/**
* The general buffer manager used by {@link InputChannel} to request/recycle exclusive or floating
* buffers.
- *
- * <p>The exclusive buffers are either entirely allocated from the local buffer pool, or they are
- * all allocated from the global pool. When allocated from the local buffer pool, the variable
- * {@code globalPool} is null.
*/
public class BufferManager implements BufferListener, BufferRecycler {
/** The available buffer queue wraps both exclusive and requested floating buffers. */
private final AvailableBufferQueue bufferQueue = new AvailableBufferQueue();
- /** The buffer provider for requesting exclusive buffers during recovery. */
- @Nullable private final MemorySegmentProvider globalPool;
+ /** The buffer provider for requesting exclusive buffers. */
+ private final MemorySegmentProvider globalPool;
/** The input channel to own this buffer manager. */
private final InputChannel inputChannel;
@@ -69,31 +66,17 @@ public class BufferManager implements BufferListener, BufferRecycler {
@GuardedBy("bufferQueue")
private boolean isWaitingForFloatingBuffers;
- private final int initialCredit;
-
- /** The number of exclusive buffers for the respective input channel. */
- @GuardedBy("bufferQueue")
- private int numExclusiveBuffers;
-
/** The total number of required buffers for the respective input channel. */
@GuardedBy("bufferQueue")
- private int numRequiredBuffers = 0;
-
- private int bufferPoolSize;
-
- public BufferManager(InputChannel inputChannel, int initialCredit) {
- this(null, inputChannel, initialCredit);
- }
+ private int numRequiredBuffers;
public BufferManager(
- @Nullable MemorySegmentProvider globalPool,
- InputChannel inputChannel,
- int initialCredit) {
- this.globalPool = globalPool;
+ MemorySegmentProvider globalPool, InputChannel inputChannel, int numRequiredBuffers) {
+
+ this.globalPool = checkNotNull(globalPool);
this.inputChannel = checkNotNull(inputChannel);
- checkArgument(initialCredit >= 0);
- this.initialCredit = initialCredit;
- this.numExclusiveBuffers = initialCredit;
+ checkArgument(numRequiredBuffers >= 0);
+ this.numRequiredBuffers = numRequiredBuffers;
}
// ------------------------------------------------------------------------
@@ -149,78 +132,15 @@ public class BufferManager implements BufferListener, BufferRecycler {
}
}
- private void resizeBufferQueue() {
- if (shouldRequestExclusiveBufferFromGlobal()) {
- return;
- }
-
- SingleInputGate inputGate = inputChannel.inputGate;
- int currentSize = inputGate.getBufferPool().getNumBuffers();
-
- // The number of local input channels can be got without lock acquired, because the number
- // is only updated when setting up or updating the channels, while it is always read after
- // these phases.
- int numRemoteChannels =
- inputGate.getNumberOfInputChannels()
- - inputGate.unsynchronizedGetNumberOfLocalInputChannels();
- if (numRemoteChannels == 0) {
- numExclusiveBuffers = 0;
- } else if (currentSize > 1 && currentSize != bufferPoolSize) {
- numExclusiveBuffers = Math.min(initialCredit, (currentSize - 1) / numRemoteChannels);
- }
- bufferPoolSize = currentSize;
- }
-
- /** Requests exclusive buffers from the local buffer pool. */
- void requestExclusiveBuffers() {
- synchronized (bufferQueue) {
- checkArgument(numExclusiveBuffers >= 0, "Num exclusive buffers must be non-negative.");
- resizeBufferQueue();
- if (numExclusiveBuffers == 0) {
- return;
- }
-
- List<MemorySegment> segments = new ArrayList<>();
- for (int i = 0; i < numExclusiveBuffers; i++) {
- BufferPool bufferPool = inputChannel.inputGate.getBufferPool();
-
- // Memory segments are requested non-blockingly to prevent getting stuck due to the
- // modification of buffer pool size by other threads. Do not worry about the lack of
- // exclusive buffers, it would be replenished with floating buffers gradually
- // upon recycling.
- MemorySegment memorySegment = bufferPool.requestMemorySegment();
- if (memorySegment != null) {
- segments.add(memorySegment);
- }
- }
-
- // AvailableBufferQueue::addExclusiveBuffer may release the previously allocated
- // floating buffer, which requires the caller to recycle these released floating
- // buffers. There should be no floating buffers that have been allocated before the
- // exclusive buffers are initialized, so here only a simple assertion is required
- checkState(
- unsynchronizedGetFloatingBuffersAvailable() == 0,
- "Bug in buffer allocation logic: floating buffer is allocated before exclusive buffers are initialized.");
- for (MemorySegment segment : segments) {
- bufferQueue.addExclusiveBuffer(
- new NetworkBuffer(segment, this), numRequiredBuffers);
- }
- }
- }
-
- /** Requests exclusive buffers from the global buffer pool. */
- void requestExclusiveBuffersFromGlobal(int numExclusiveBuffers) throws IOException {
- checkState(numExclusiveBuffers >= 0, "Num exclusive buffers must be non-negative.");
- checkState(
- shouldRequestExclusiveBufferFromGlobal(),
- "Only used when requesting buffers from global buffer pool.");
+ /** Requests exclusive buffers from the provider. */
+ void requestExclusiveBuffers(int numExclusiveBuffers) throws IOException {
+ checkArgument(numExclusiveBuffers >= 0, "Num exclusive buffers must be non-negative.");
if (numExclusiveBuffers == 0) {
return;
}
- List<MemorySegment> segments = new ArrayList<>();
- segments.addAll(globalPool.requestUnpooledMemorySegments(numExclusiveBuffers));
-
+ Collection<MemorySegment> segments =
+ globalPool.requestUnpooledMemorySegments(numExclusiveBuffers);
synchronized (bufferQueue) {
// AvailableBufferQueue::addExclusiveBuffer may release the previously allocated
// floating buffer, which requires the caller to recycle these released floating
@@ -276,16 +196,6 @@ public class BufferManager implements BufferListener, BufferRecycler {
return numRequestedBuffers;
}
- /**
- * The {@link LocalRecoveredInputChannel} also needs buffers to store the state, however, the
- * expected size of local buffer pool is calculated with the number of remote input channels. So
- * we request exclusive buffers for the {@link LocalRecoveredInputChannel} from the global
- * buffer pool and these buffers are released once the recovery is finish.
- */
- private boolean shouldRequestExclusiveBufferFromGlobal() {
- return globalPool != null;
- }
-
// ------------------------------------------------------------------------
// Buffer recycle
// ------------------------------------------------------------------------
@@ -300,22 +210,12 @@ public class BufferManager implements BufferListener, BufferRecycler {
public void recycle(MemorySegment segment) {
@Nullable Buffer releasedFloatingBuffer = null;
synchronized (bufferQueue) {
- resizeBufferQueue();
try {
- BufferPool bufferPool = inputChannel.inputGate.getBufferPool();
// Similar to notifyBufferAvailable(), make sure that we never add a buffer
// after channel released all buffers via releaseAllResources().
if (inputChannel.isReleased()) {
- if (shouldRequestExclusiveBufferFromGlobal()) {
- globalPool.recycleUnpooledMemorySegments(
- Collections.singletonList(segment));
- } else {
- bufferPool.recycle(segment);
- }
+ globalPool.recycleUnpooledMemorySegments(Collections.singletonList(segment));
return;
- } else if (bufferQueue.exclusiveBuffers.size() >= numExclusiveBuffers
- && !shouldRequestExclusiveBufferFromGlobal()) {
- bufferPool.recycle(segment);
} else {
releasedFloatingBuffer =
bufferQueue.addExclusiveBuffer(
@@ -354,17 +254,15 @@ public class BufferManager implements BufferListener, BufferRecycler {
/** Recycles all the exclusive and floating buffers from the given buffer queue. */
void releaseAllBuffers(ArrayDeque<Buffer> buffers) throws IOException {
- // For the exclusive buffers that are requested from global pool, gather them all and
- // recycle to global pool in batch, because we do not want to trigger redistribution of
- // buffers after each recycle.
+ // Gather all exclusive buffers and recycle them to global pool in batch, because
+ // we do not want to trigger redistribution of buffers after each recycle.
final List<MemorySegment> exclusiveRecyclingSegments = new ArrayList<>();
Exception err = null;
Buffer buffer;
while ((buffer = buffers.poll()) != null) {
try {
- if (buffer.getRecycler() == BufferManager.this
- && shouldRequestExclusiveBufferFromGlobal()) {
+ if (buffer.getRecycler() == BufferManager.this) {
exclusiveRecyclingSegments.add(buffer.getMemorySegment());
} else {
buffer.recycleBuffer();
@@ -375,12 +273,7 @@ public class BufferManager implements BufferListener, BufferRecycler {
}
try {
synchronized (bufferQueue) {
- if (shouldRequestExclusiveBufferFromGlobal()) {
- bufferQueue.releaseAll(exclusiveRecyclingSegments);
- } else {
- numExclusiveBuffers = 0;
- bufferQueue.releaseAll(null);
- }
+ bufferQueue.releaseAll(exclusiveRecyclingSegments);
bufferQueue.notifyAll();
}
} catch (Exception e) {
@@ -493,12 +386,6 @@ public class BufferManager implements BufferListener, BufferRecycler {
}
}
- int getNumExclusiveBuffers() {
- synchronized (bufferQueue) {
- return numExclusiveBuffers;
- }
- }
-
int unsynchronizedGetAvailableExclusiveBuffers() {
return bufferQueue.exclusiveBuffers.size();
}
@@ -565,22 +452,17 @@ public class BufferManager implements BufferListener, BufferRecycler {
/**
* The floating buffer is recycled to local buffer pool directly, and the exclusive buffer
- * that is requested from global buffer pool will be gathered to return to global buffer
- * pool later.
+ * will be gathered to return to global buffer pool later.
*
* @param exclusiveSegments The list that we will add exclusive segments into.
*/
- void releaseAll(@Nullable List<MemorySegment> exclusiveSegments) {
+ void releaseAll(List<MemorySegment> exclusiveSegments) {
Buffer buffer;
while ((buffer = floatingBuffers.poll()) != null) {
buffer.recycleBuffer();
}
while ((buffer = exclusiveBuffers.poll()) != null) {
- if (exclusiveSegments != null) {
- exclusiveSegments.add(buffer.getMemorySegment());
- } else {
- buffer.recycleBuffer();
- }
+ exclusiveSegments.add(buffer.getMemorySegment());
}
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
index 661843cd096..f415ff6165f 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
@@ -98,13 +98,11 @@ public class InputGateSpecUtils {
int configuredNetworkBuffersPerChannel,
int numInputChannels,
int requiredBuffersPerGate) {
+ checkArgument(numInputChannels > 0, "Must be positive.");
checkArgument(requiredBuffersPerGate >= 1, "Require at least 1 buffer per gate.");
-
- return numInputChannels == 0
- ? 0
- : Math.min(
- configuredNetworkBuffersPerChannel,
- (requiredBuffersPerGate - 1) / numInputChannels);
+ return Math.min(
+ configuredNetworkBuffersPerChannel,
+ (requiredBuffersPerGate - 1) / numInputChannels);
}
private static int getExpectedBuffersTargetPerGate(
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalRecoveredInputChannel.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalRecoveredInputChannel.java
index e3213b37d13..784444d63e8 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalRecoveredInputChannel.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalRecoveredInputChannel.java
@@ -19,14 +19,11 @@
package org.apache.flink.runtime.io.network.partition.consumer;
import org.apache.flink.runtime.io.network.TaskEventPublisher;
-import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.metrics.InputChannelMetrics;
import org.apache.flink.runtime.io.network.partition.ResultPartitionID;
import org.apache.flink.runtime.io.network.partition.ResultPartitionManager;
import org.apache.flink.runtime.io.network.partition.ResultSubpartitionIndexSet;
-import java.io.IOException;
-
import static org.apache.flink.util.Preconditions.checkNotNull;
/**
@@ -36,7 +33,6 @@ import static org.apache.flink.util.Preconditions.checkNotNull;
public class LocalRecoveredInputChannel extends RecoveredInputChannel {
private final ResultPartitionManager partitionManager;
private final TaskEventPublisher taskEventPublisher;
- private boolean exclusiveBuffersAssigned;
LocalRecoveredInputChannel(
SingleInputGate inputGate,
@@ -62,7 +58,6 @@ public class LocalRecoveredInputChannel extends RecoveredInputChannel {
this.partitionManager = checkNotNull(partitionManager);
this.taskEventPublisher = checkNotNull(taskEventPublisher);
- this.bufferManager = new BufferManager(inputGate.getMemorySegmentProvider(), this, 0);
}
@Override
@@ -80,14 +75,4 @@ public class LocalRecoveredInputChannel extends RecoveredInputChannel {
numBuffersIn,
channelStateWriter);
}
-
- @Override
- public Buffer requestBufferBlocking() throws InterruptedException, IOException {
- // not in setup to avoid assigning buffers unnecessarily if there is no state
- if (!exclusiveBuffersAssigned) {
- bufferManager.requestExclusiveBuffersFromGlobal(networkBuffersPerChannel);
- exclusiveBuffersAssigned = true;
- }
- return super.requestBufferBlocking();
- }
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RecoveredInputChannel.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RecoveredInputChannel.java
index b05abbaab71..1f41a099931 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RecoveredInputChannel.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RecoveredInputChannel.java
@@ -55,7 +55,7 @@ public abstract class RecoveredInputChannel extends InputChannel implements Chan
private final ArrayDeque<Buffer> receivedBuffers = new ArrayDeque<>();
private final CompletableFuture<?> stateConsumedFuture = new CompletableFuture<>();
- protected BufferManager bufferManager;
+ protected final BufferManager bufferManager;
@GuardedBy("receivedBuffers")
private boolean isReleased;
@@ -69,6 +69,7 @@ public abstract class RecoveredInputChannel extends InputChannel implements Chan
private int sequenceNumber = Integer.MIN_VALUE;
protected final int networkBuffersPerChannel;
+ private boolean exclusiveBuffersAssigned;
private long lastStoppedCheckpointId = -1;
@@ -91,6 +92,8 @@ public abstract class RecoveredInputChannel extends InputChannel implements Chan
maxBackoff,
numBytesIn,
numBuffersIn);
+
+ bufferManager = new BufferManager(inputGate.getMemorySegmentProvider(), this, 0);
this.networkBuffersPerChannel = networkBuffersPerChannel;
}
@@ -272,6 +275,11 @@ public abstract class RecoveredInputChannel extends InputChannel implements Chan
}
public Buffer requestBufferBlocking() throws InterruptedException, IOException {
+ // not in setup to avoid assigning buffers unnecessarily if there is no state
+ if (!exclusiveBuffersAssigned) {
+ bufferManager.requestExclusiveBuffers(networkBuffersPerChannel);
+ exclusiveBuffersAssigned = true;
+ }
return bufferManager.requestBufferBlocking();
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
index 11e2d3620f5..e41d209b4ea 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java
@@ -155,7 +155,7 @@ public class RemoteInputChannel extends InputChannel {
this.initialCredit = networkBuffersPerChannel;
this.connectionId = checkNotNull(connectionId);
this.connectionManager = checkNotNull(connectionManager);
- this.bufferManager = new BufferManager(this, initialCredit);
+ this.bufferManager = new BufferManager(inputGate.getMemorySegmentProvider(), this, 0);
this.channelStatePersister = new ChannelStatePersister(stateWriter, getChannelInfo());
}
@@ -174,7 +174,7 @@ public class RemoteInputChannel extends InputChannel {
bufferManager.unsynchronizedGetAvailableExclusiveBuffers() == 0,
"Bug in input channel setup logic: exclusive buffers have already been set for this input channel.");
- bufferManager.requestExclusiveBuffers();
+ bufferManager.requestExclusiveBuffers(initialCredit);
}
// ------------------------------------------------------------------------
@@ -343,8 +343,7 @@ public class RemoteInputChannel extends InputChannel {
@Override
int getBuffersInUseCount() {
return getNumberOfQueuedBuffers()
- + Math.max(
- 0, bufferManager.getNumberOfRequiredBuffers() - getNumExclusiveBuffers());
+ + Math.max(0, bufferManager.getNumberOfRequiredBuffers() - initialCredit);
}
@Override
@@ -397,7 +396,7 @@ public class RemoteInputChannel extends InputChannel {
@VisibleForTesting
public int getSenderBacklog() {
- return getNumberOfRequiredBuffers() - getNumExclusiveBuffers();
+ return getNumberOfRequiredBuffers() - initialCredit;
}
@VisibleForTesting
@@ -437,7 +436,7 @@ public class RemoteInputChannel extends InputChannel {
checkState(!isReleased.get(), "Channel released.");
checkPartitionRequestQueueInitialized();
- if (getNumExclusiveBuffers() == 0) {
+ if (initialCredit == 0) {
// this unannounced credit can be a positive value because credit assignment and the
// increase of this value is not an atomic operation and as a result, this unannounced
// credit value can be get increased even after this channel has been blocked and all
@@ -460,7 +459,7 @@ public class RemoteInputChannel extends InputChannel {
}
private void onBlockingUpstream() {
- if (getNumExclusiveBuffers() == 0) {
+ if (initialCredit == 0) {
// release the allocated floating buffers so that they can be used by other channels if
// no exclusive buffer is configured, it is important because a blocked channel can not
// transmit any data so the allocated floating buffers can not be recycled, as a result,
@@ -515,23 +514,21 @@ public class RemoteInputChannel extends InputChannel {
public int unsynchronizedGetExclusiveBuffersUsed() {
return Math.max(
- 0,
- getNumExclusiveBuffers()
- - bufferManager.unsynchronizedGetAvailableExclusiveBuffers());
+ 0, initialCredit - bufferManager.unsynchronizedGetAvailableExclusiveBuffers());
}
public int unsynchronizedGetFloatingBuffersAvailable() {
return Math.max(0, bufferManager.unsynchronizedGetFloatingBuffersAvailable());
}
- public int getNumExclusiveBuffers() {
- return bufferManager.getNumExclusiveBuffers();
- }
-
public InputChannelID getInputChannelId() {
return id;
}
+ public int getInitialCredit() {
+ return initialCredit;
+ }
+
public BufferProvider getBufferProvider() throws IOException {
if (isReleased.get()) {
return null;
@@ -553,14 +550,13 @@ public class RemoteInputChannel extends InputChannel {
/**
* Receives the backlog from the producer's buffer response. If the number of available buffers
- * is less than backlog + numExclusiveBuffers, it will request floating buffers from the buffer
+ * is less than backlog + initialCredit, it will request floating buffers from the buffer
* manager, and then notify unannounced credits to the producer.
*
* @param backlog The number of unsent buffers in the producer's sub partition.
*/
public void onSenderBacklog(int backlog) throws IOException {
- notifyBufferAvailable(
- bufferManager.requestFloatingBuffers(backlog + getNumExclusiveBuffers()));
+ notifyBufferAvailable(bufferManager.requestFloatingBuffers(backlog + initialCredit));
}
/**
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteRecoveredInputChannel.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteRecoveredInputChannel.java
index 183457df34b..cbaddbcbfa0 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteRecoveredInputChannel.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteRecoveredInputChannel.java
@@ -63,7 +63,6 @@ public class RemoteRecoveredInputChannel extends RecoveredInputChannel {
this.connectionId = checkNotNull(connectionId);
this.connectionManager = checkNotNull(connectionManager);
this.partitionRequestListenerTimeout = partitionRequestListenerTimeout;
- this.bufferManager = new BufferManager(this, networkBuffersPerChannel);
}
@Override
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
index 4fc8416a8e9..ff0207acf93 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java
@@ -150,10 +150,6 @@ public class SingleInputGate extends IndexedInputGate {
/** The number of input channels (equivalent to the number of consumed partitions). */
private final int numberOfInputChannels;
- /** The number of local input channels. */
- @GuardedBy("requestLock")
- private int numberOfLocalInputChannels;
-
/** Input channels. We store this in a map for runtime updates of single channels. */
private final Map<IntermediateResultPartitionID, Map<InputChannelInfo, InputChannel>>
inputChannels;
@@ -542,10 +538,6 @@ public class SingleInputGate extends IndexedInputGate {
return 0;
}
- public int unsynchronizedGetNumberOfLocalInputChannels() {
- return numberOfLocalInputChannels;
- }
-
public CompletableFuture<Void> getCloseFuture() {
return closeFuture;
}
@@ -609,10 +601,6 @@ public class SingleInputGate extends IndexedInputGate {
numberOfUninitializedChannels++;
}
- if (inputChannel instanceof LocalInputChannel
- || inputChannel instanceof LocalRecoveredInputChannel) {
- numberOfLocalInputChannels++;
- }
}
}
}
@@ -652,7 +640,6 @@ public class SingleInputGate extends IndexedInputGate {
newChannel =
unknownChannel.toLocalInputChannel(
shuffleDescriptor.getResultPartitionID());
- numberOfLocalInputChannels++;
} else {
RemoteInputChannel remoteInputChannel =
unknownChannel.toRemoteInputChannel(
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
index 35700a2c460..17669ac81f6 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
@@ -163,7 +163,10 @@ public class SingleInputGateFactory {
configuredNetworkBuffersPerChannel,
floatingNetworkBuffersPerGate,
igdd.getConsumedPartitionType(),
- calculateNumNonLocalChannels(igdd, isSharedInputChannelSupported),
+ calculateNumChannels(
+ igdd.getShuffleDescriptors().length,
+ igdd.getConsumedSubpartitionIndexRange().size(),
+ isSharedInputChannelSupported),
tieredStorageConfiguration != null);
SupplierWithException<BufferPool, IOException> bufferPoolFactory =
createBufferPoolFactory(
@@ -380,26 +383,6 @@ public class SingleInputGateFactory {
}
}
- private int calculateNumNonLocalChannels(
- @Nonnull InputGateDeploymentDescriptor igdd, boolean isSharedInputChannelSupported) {
- int count = 0;
- for (ShuffleDescriptor shuffleDescriptor : igdd.getShuffleDescriptors()) {
- boolean isLocal =
- applyWithShuffleTypeCheck(
- NettyShuffleDescriptor.class,
- shuffleDescriptor,
- unknownShuffleDescriptor -> false,
- nettyShuffleDescriptor -> isLocalInputChannel(nettyShuffleDescriptor));
- if (!isLocal) {
- count +=
- isSharedInputChannelSupported
- ? 1
- : igdd.getConsumedSubpartitionIndexRange().size();
- }
- }
- return count;
- }
-
@VisibleForTesting
protected InputChannel createKnownInputChannel(
SingleInputGate inputGate,
@@ -410,7 +393,7 @@ public class SingleInputGateFactory {
ChannelStatistics channelStatistics,
InputChannelMetrics metrics) {
ResultPartitionID partitionId = inputChannelDescriptor.getResultPartitionID();
- if (isLocalInputChannel(inputChannelDescriptor)) {
+ if (inputChannelDescriptor.isLocalTo(taskExecutorResourceId)) {
// Consuming task is deployed to the same TaskManager as the partition => local
channelStatistics.numLocalChannels++;
return new LocalRecoveredInputChannel(
@@ -442,10 +425,6 @@ public class SingleInputGateFactory {
}
}
- private boolean isLocalInputChannel(NettyShuffleDescriptor inputChannelDescriptor) {
- return inputChannelDescriptor.isLocalTo(taskExecutorResourceId);
- }
-
private void addTierShuffleDescriptors(
List<List<TierShuffleDescriptor>> tierShuffleDescriptors,
ShuffleDescriptor descriptor) {
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/InputChannelRecoveredStateHandlerTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/InputChannelRecoveredStateHandlerTest.java
index 646bfb39815..823183be99e 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/InputChannelRecoveredStateHandlerTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/InputChannelRecoveredStateHandlerTest.java
@@ -30,7 +30,6 @@ import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateBui
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
-import java.io.IOException;
import java.util.HashSet;
import static org.assertj.core.api.Assertions.assertThat;
@@ -44,7 +43,7 @@ class InputChannelRecoveredStateHandlerTest extends RecoveredChannelStateHandler
private InputChannelInfo channelInfo;
@BeforeEach
- void setUp() throws IOException {
+ void setUp() {
// given: Segment provider with defined number of allocated segments.
networkBufferPool = new NetworkBufferPool(preAllocatedSegments, 1024);
@@ -53,10 +52,7 @@ class InputChannelRecoveredStateHandlerTest extends RecoveredChannelStateHandler
new SingleInputGateBuilder()
.setChannelFactory(InputChannelBuilder::buildLocalRecoveredChannel)
.setSegmentProvider(networkBufferPool)
- .setBufferPoolFactory(
- networkBufferPool.createBufferPool(1, 1, preAllocatedSegments))
.build();
- inputGate.setup();
icsHandler = buildInputChannelStateHandler(inputGate);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java
index ebbd15e710d..b7d7c192c9c 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java
@@ -211,7 +211,7 @@ public class SequentialChannelStateReaderImplTest {
.setSingleInputGateIndex(i)
.setBufferPoolFactory(
networkBufferPool.createBufferPool(
- 1, 1, parLevel * buffersPerChannel))
+ 1, 1, buffersPerChannel))
.setSegmentProvider(networkBufferPool)
.setChannelFactory(
(builder, gate) ->
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java
index dee969cb051..3d18d22eee4 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java
@@ -105,7 +105,8 @@ class NettyShuffleEnvironmentTest {
// outgoing: 1 buffer per channel + 1 extra buffer per ResultPartition
// incoming: 2 exclusive buffers per channel + 1 floating buffer per single gate
final int bufferCount =
- 8 + 10 * NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_PER_CHANNEL.defaultValue();
+ 18 + 10 * NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_PER_CHANNEL.defaultValue();
+
testRegisterTaskWithLimitedBuffers(bufferCount);
}
@@ -118,7 +119,7 @@ class NettyShuffleEnvironmentTest {
// outgoing: 1 buffer per channel + 1 extra buffer per ResultPartition
// incoming: 2 exclusive buffers per channel + 1 floating buffer per single gate
final int bufferCount =
- 8
+ 10
+ 10
* NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_PER_CHANNEL
.defaultValue()
@@ -327,14 +328,10 @@ class NettyShuffleEnvironmentTest {
// verify buffer pools for the input gates (NOTE: credit-based uses minimum required buffers
// for exclusive buffers not managed by the buffer pool)
- assertThat(ig1.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(ig1.getNumberOfInputChannels() + 1);
- assertThat(ig2.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(ig2.getNumberOfInputChannels() + 1);
- assertThat(ig3.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(ig3.getNumberOfInputChannels() + 1);
- assertThat(ig4.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(ig4.getNumberOfInputChannels() + 1);
+ assertThat(ig1.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
+ assertThat(ig2.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
+ assertThat(ig3.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
+ assertThat(ig4.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
assertThat(ig1.getBufferPool().getMaxNumberOfMemorySegments()).isEqualTo(floatingBuffers);
assertThat(ig2.getBufferPool().getMaxNumberOfMemorySegments()).isEqualTo(floatingBuffers);
@@ -373,7 +370,7 @@ class NettyShuffleEnvironmentTest {
new SingleInputGateBuilder()
.setNumberOfChannels(numberOfChannels)
.setResultPartitionType(partitionType)
- .setupBufferPoolFactory(network, numberOfChannels + 1)
+ .setupBufferPoolFactory(network)
.build());
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NoOpBufferPool.java
similarity index 90%
rename from flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java
rename to flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NoOpBufferPool.java
index e34c4bb8dac..bb5189215c7 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/TestingBufferPool.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NoOpBufferPool.java
@@ -21,12 +21,11 @@
package org.apache.flink.runtime.io.network.buffer;
import org.apache.flink.core.memory.MemorySegment;
-import org.apache.flink.core.memory.MemorySegmentFactory;
import java.util.concurrent.CompletableFuture;
-/** Implementation of {@link BufferPool} for testing. */
-public class TestingBufferPool implements BufferPool {
+/** No-op implementation of {@link BufferPool}. */
+public class NoOpBufferPool implements BufferPool {
@Override
public void reserveSegments(int numberOfSegmentsToReserve) {}
@@ -36,8 +35,7 @@ public class TestingBufferPool implements BufferPool {
@Override
public Buffer requestBuffer() {
- MemorySegment segment = requestMemorySegment();
- return new NetworkBuffer(segment, FreeingBufferRecycler.INSTANCE);
+ throw new UnsupportedOperationException();
}
@Override
@@ -73,7 +71,7 @@ public class TestingBufferPool implements BufferPool {
@Override
public MemorySegment requestMemorySegment() {
- return MemorySegmentFactory.allocateUnpooledSegment(1024);
+ throw new UnsupportedOperationException();
}
@Override
@@ -98,7 +96,7 @@ public class TestingBufferPool implements BufferPool {
@Override
public int getNumBuffers() {
- return 0;
+ throw new UnsupportedOperationException();
}
@Override
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
index 7d78fba72bb..ef83f0ce7f3 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
@@ -32,7 +32,6 @@ import org.apache.flink.runtime.io.network.buffer.BufferListener;
import org.apache.flink.runtime.io.network.buffer.BufferPool;
import org.apache.flink.runtime.io.network.buffer.BufferProvider;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
-import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
import org.apache.flink.runtime.io.network.netty.NettyMessage.AddCredit;
import org.apache.flink.runtime.io.network.netty.NettyMessage.BufferResponse;
import org.apache.flink.runtime.io.network.netty.NettyMessage.CloseRequest;
@@ -263,8 +262,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
inputGate.setInputChannels(inputChannel);
try {
- BufferPool bufferPool =
- networkBufferPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(8, 8, 8);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -397,16 +395,14 @@ class CreditBasedPartitionRequestClientHandlerTest {
mock(ConnectionID.class),
mock(PartitionRequestClientFactory.class));
- final int numBuffers = 10;
- final NetworkBufferPool networkBufferPool = new NetworkBufferPool(numBuffers, 32);
+ final NetworkBufferPool networkBufferPool = new NetworkBufferPool(10, 32);
final SingleInputGate inputGate = createSingleInputGate(2, networkBufferPool);
final RemoteInputChannel[] inputChannels = new RemoteInputChannel[2];
inputChannels[0] = createRemoteInputChannel(inputGate, client);
inputChannels[1] = createRemoteInputChannel(inputGate, client);
try {
inputGate.setInputChannels(inputChannels);
- final BufferPool bufferPool =
- networkBufferPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -595,7 +591,6 @@ class CreditBasedPartitionRequestClientHandlerTest {
final String expectedMessage = "test exception on buffer";
final NetworkBufferPool networkBufferPool = new NetworkBufferPool(10, bufferSize);
final SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
- inputGate.setBufferPool(new TestingBufferPool());
final RemoteInputChannel inputChannel =
new TestRemoteInputChannelForError(inputGate, expectedMessage);
final CreditBasedPartitionRequestClientHandler handler =
@@ -603,7 +598,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
try {
inputGate.setInputChannels(inputChannel);
- inputGate.setupChannels();
+ inputGate.setup();
inputGate.requestPartitions();
handler.addInputChannel(inputChannel);
@@ -725,16 +720,12 @@ class CreditBasedPartitionRequestClientHandlerTest {
boolean isRemoved, boolean readBeforeReleasingOrRemoving) throws Exception {
int bufferSize = 1024;
- int numBuffers = 10;
- NetworkBufferPool networkBufferPool = new NetworkBufferPool(numBuffers, bufferSize);
+ NetworkBufferPool networkBufferPool = new NetworkBufferPool(10, bufferSize);
SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
- final BufferPool bufferPool =
- networkBufferPool.createBufferPool(numBuffers, numBuffers, numBuffers);
- inputGate.setBufferPool(bufferPool);
RemoteInputChannel inputChannel = new InputChannelBuilder().buildRemoteChannel(inputGate);
inputGate.setInputChannels(inputChannel);
- inputGate.setupChannels();
+ inputGate.setup();
CreditBasedPartitionRequestClientHandler handler =
new CreditBasedPartitionRequestClientHandler();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java
index 96611550686..47e088aa69e 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientDecoderDelegateTest.java
@@ -25,7 +25,6 @@ import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
-import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
import org.apache.flink.runtime.io.network.partition.consumer.InputChannelBuilder;
import org.apache.flink.runtime.io.network.partition.consumer.InputChannelID;
import org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel;
@@ -78,12 +77,11 @@ class NettyMessageClientDecoderDelegateTest {
channel = new EmbeddedChannel(new NettyMessageClientDecoderDelegate(handler));
inputGate = createSingleInputGate(1, networkBufferPool);
- inputGate.setBufferPool(new TestingBufferPool());
RemoteInputChannel inputChannel =
createRemoteInputChannel(
inputGate, new TestingPartitionRequestClient(), NUMBER_OF_BUFFER_RESPONSES);
inputGate.setInputChannels(inputChannel);
- inputGate.setupChannels();
+ inputGate.setup();
inputChannel.requestSubpartitions();
handler.addInputChannel(inputChannel);
inputChannelId = inputChannel.getInputChannelId();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java
index 8f4123a08a6..b30981c7362 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageClientSideSerializationTest.java
@@ -28,7 +28,6 @@ import org.apache.flink.runtime.io.network.buffer.BufferDecompressor;
import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
-import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
import org.apache.flink.runtime.io.network.partition.consumer.InputChannelID;
import org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel;
import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate;
@@ -81,12 +80,11 @@ class NettyMessageClientSideSerializationTest {
void setup() throws IOException, InterruptedException {
networkBufferPool = new NetworkBufferPool(8, BUFFER_SIZE);
inputGate = createSingleInputGate(1, networkBufferPool);
- inputGate.setBufferPool(new TestingBufferPool());
RemoteInputChannel inputChannel =
createRemoteInputChannel(inputGate, new TestingPartitionRequestClient());
inputChannel.requestSubpartitions();
inputGate.setInputChannels(inputChannel);
- inputGate.setupChannels();
+ inputGate.setup();
CreditBasedPartitionRequestClientHandler handler =
new CreditBasedPartitionRequestClientHandler();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
index 419bf508d8e..5592237b98a 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/InputGateFairnessTest.java
@@ -24,7 +24,7 @@ import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.buffer.BufferConsumer;
import org.apache.flink.runtime.io.network.buffer.BufferPool;
-import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
+import org.apache.flink.runtime.io.network.buffer.NoOpBufferPool;
import org.apache.flink.runtime.io.network.partition.InputChannelTestUtils.UnpooledMemorySegmentProvider;
import org.apache.flink.runtime.io.network.partition.consumer.BufferOrEvent;
import org.apache.flink.runtime.io.network.partition.consumer.InputChannel;
@@ -227,6 +227,7 @@ public class InputGateFairnessTest {
}
gate.setInputChannels(channels);
+ gate.setup();
gate.requestPartitions();
// read all the buffers and the EOF event
@@ -272,10 +273,7 @@ public class InputGateFairnessTest {
channels[11].onBuffer(mockBuffer, 0, -1, 0);
channelSequenceNums[11]++;
- gate.setBufferPool(new TestingBufferPool());
- gate.setInputChannels(channels);
- gate.setupChannels();
- gate.requestPartitions();
+ setupInputGate(gate, channels);
// read all the buffers and the EOF event
for (int i = 0; i < numberOfChannels * buffersPerChannel; i++) {
@@ -353,7 +351,7 @@ public class InputGateFairnessTest {
private static class FairnessVerifyingInputGate extends SingleInputGate {
private static final int BUFFER_SIZE = 32 * 1024;
private static final SupplierWithException<BufferPool, IOException>
- STUB_BUFFER_POOL_FACTORY = TestingBufferPool::new;
+ STUB_BUFFER_POOL_FACTORY = NoOpBufferPool::new;
private final PrioritizedDeque<InputChannel> channelsWithData;
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PartitionTestUtils.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PartitionTestUtils.java
index fa216502118..994e7295e19 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PartitionTestUtils.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PartitionTestUtils.java
@@ -26,7 +26,6 @@ import org.apache.flink.runtime.io.network.NettyShuffleEnvironment;
import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
-import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
import org.apache.flink.runtime.shuffle.PartitionDescriptor;
import org.apache.flink.runtime.shuffle.PartitionDescriptorBuilder;
import org.apache.flink.runtime.shuffle.ShuffleDescriptor;
@@ -75,21 +74,6 @@ public enum PartitionTestUtils {
.build();
}
- public static ResultPartition createPartition(
- ResultPartitionType type,
- FileChannelManager channelManager,
- NetworkBufferPool networkBufferPool,
- boolean compressionEnabled,
- int networkBufferSize) {
- return new ResultPartitionBuilder()
- .setResultPartitionType(type)
- .setFileChannelManager(channelManager)
- .setNetworkBufferPool(networkBufferPool)
- .setBlockingShuffleCompressionEnabled(compressionEnabled)
- .setNetworkBufferSize(networkBufferSize)
- .build();
- }
-
public static ResultPartition createPartition(
NettyShuffleEnvironment environment,
ResultPartitionType partitionType,
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartitionWithReadViewTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartitionWithReadViewTest.java
index dd6f3ab81f8..e562e73c672 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartitionWithReadViewTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartitionWithReadViewTest.java
@@ -31,7 +31,6 @@ import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.buffer.BufferBuilder;
import org.apache.flink.runtime.io.network.buffer.BufferConsumer;
-import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
import org.apache.flink.runtime.state.CheckpointStorageLocationReference;
import org.apache.flink.testutils.junit.extensions.parameterized.Parameter;
import org.apache.flink.testutils.junit.extensions.parameterized.ParameterizedTestExtension;
@@ -671,12 +670,10 @@ class PipelinedSubpartitionWithReadViewTest {
}
void setup(ResultPartitionType resultPartitionType) throws IOException {
- NetworkBufferPool networkBufferPool = new NetworkBufferPool(4, BUFFER_SIZE);
resultPartition =
PartitionTestUtils.createPartition(
resultPartitionType,
NoOpFileChannelManager.INSTANCE,
- networkBufferPool,
compressionEnabled,
BUFFER_SIZE);
resultPartition.setup();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java
index b53a23ca93f..103ec3da8bf 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java
@@ -58,7 +58,7 @@ public class ResultPartitionBuilder {
private FileChannelManager channelManager = NoOpFileChannelManager.INSTANCE;
- private NetworkBufferPool networkBufferPool = new NetworkBufferPool(4, 1);
+ private NetworkBufferPool networkBufferPool = new NetworkBufferPool(2, 1);
private BatchShuffleReadBufferPool batchShuffleReadBufferPool =
new BatchShuffleReadBufferPool(64 * 32 * 1024, 32 * 1024);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java
index 05fdd8c086f..8e7b79f92dd 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java
@@ -292,12 +292,10 @@ class ResultPartitionTest {
try {
resultPartition.setup();
- resultPartition.getBufferPool().setNumBuffers(4);
+ resultPartition.getBufferPool().setNumBuffers(2);
assertThat(resultPartition.getAvailableFuture()).isDone();
- resultPartition.emitRecord(ByteBuffer.allocate(bufferSize), 0);
- resultPartition.emitRecord(ByteBuffer.allocate(bufferSize), 0);
resultPartition.emitRecord(ByteBuffer.allocate(bufferSize), 0);
resultPartition.emitRecord(ByteBuffer.allocate(bufferSize), 0);
assertThat(resultPartition.getAvailableFuture()).isNotDone();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/InputBuffersMetricsTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/InputBuffersMetricsTest.java
index 4e2d79d7fdc..f512aee9f9b 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/InputBuffersMetricsTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/InputBuffersMetricsTest.java
@@ -78,13 +78,13 @@ class InputBuffersMetricsTest {
inputGate1.setup();
SingleInputGate[] inputGates = new SingleInputGate[] {inputGate1};
- CreditBasedInputBuffersUsageGauge inputBufferPoolUsageGauge =
- new CreditBasedInputBuffersUsageGauge(inputGates);
+ FloatingBuffersUsageGauge floatingBuffersUsageGauge =
+ new FloatingBuffersUsageGauge(inputGates);
ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge =
new ExclusiveBuffersUsageGauge(inputGates);
- FloatingBuffersUsageGauge floatingBuffersUsageGauge =
- new FloatingBuffersUsageGauge(
- inputGates, inputBufferPoolUsageGauge, exclusiveBuffersUsageGauge);
+ CreditBasedInputBuffersUsageGauge inputBufferPoolUsageGauge =
+ new CreditBasedInputBuffersUsageGauge(
+ floatingBuffersUsageGauge, exclusiveBuffersUsageGauge, inputGates);
closeableRegistry.registerCloseable(network::close);
closeableRegistry.registerCloseable(inputGate1::close);
@@ -132,10 +132,13 @@ class InputBuffersMetricsTest {
List<RemoteInputChannel> remoteInputChannels = tuple1.f1;
SingleInputGate[] inputGates = new SingleInputGate[] {tuple1.f0, tuple2.f0};
- CreditBasedInputBuffersUsageGauge inputBuffersUsageGauge =
- new CreditBasedInputBuffersUsageGauge(inputGates);
+ FloatingBuffersUsageGauge floatingBuffersUsageGauge =
+ new FloatingBuffersUsageGauge(inputGates);
ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge =
new ExclusiveBuffersUsageGauge(inputGates);
+ CreditBasedInputBuffersUsageGauge inputBuffersUsageGauge =
+ new CreditBasedInputBuffersUsageGauge(
+ floatingBuffersUsageGauge, exclusiveBuffersUsageGauge, inputGates);
assertThat(exclusiveBuffersUsageGauge.getValue()).isEqualTo(0.0f, offset(0.0f));
assertThat(inputBuffersUsageGauge.getValue()).isEqualTo(0.0f, offset(0.0f));
@@ -192,13 +195,13 @@ class InputBuffersMetricsTest {
RemoteInputChannel remoteInputChannel1 = tuple1.f1.get(0);
SingleInputGate[] inputGates = new SingleInputGate[] {tuple1.f0, inputGate2};
- CreditBasedInputBuffersUsageGauge inputBuffersUsageGauge =
- new CreditBasedInputBuffersUsageGauge(inputGates);
+ FloatingBuffersUsageGauge floatingBuffersUsageGauge =
+ new FloatingBuffersUsageGauge(inputGates);
ExclusiveBuffersUsageGauge exclusiveBuffersUsageGauge =
new ExclusiveBuffersUsageGauge(inputGates);
- FloatingBuffersUsageGauge floatingBuffersUsageGauge =
- new FloatingBuffersUsageGauge(
- inputGates, inputBuffersUsageGauge, exclusiveBuffersUsageGauge);
+ CreditBasedInputBuffersUsageGauge inputBuffersUsageGauge =
+ new CreditBasedInputBuffersUsageGauge(
+ floatingBuffersUsageGauge, exclusiveBuffersUsageGauge, inputGates);
assertThat(floatingBuffersUsageGauge.getValue()).isEqualTo(0.0f, offset(0.0f));
assertThat(inputBuffersUsageGauge.getValue()).isEqualTo(0.0f, offset(0.0f));
@@ -269,14 +272,7 @@ class InputBuffersMetricsTest {
new SingleInputGateBuilder()
.setNumberOfChannels(numberOfRemoteChannels + numberOfLocalChannels)
.setResultPartitionType(ResultPartitionType.PIPELINED_BOUNDED)
- .setupBufferPoolFactory(
- network,
- 1,
- numberOfRemoteChannels
- * network.getConfiguration()
- .networkBuffersPerChannel()
- + network.getConfiguration()
- .floatingNetworkBuffersPerGate())
+ .setupBufferPoolFactory(network)
.build();
InputChannel[] inputChannels =
new InputChannel[numberOfRemoteChannels + numberOfLocalChannels];
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
index 6abadbf10ea..8d5448c8679 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
@@ -18,6 +18,8 @@
package org.apache.flink.runtime.io.network.partition.consumer;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
import org.apache.flink.core.testutils.OneShotLatch;
import org.apache.flink.runtime.checkpoint.CheckpointException;
import org.apache.flink.runtime.checkpoint.CheckpointOptions;
@@ -39,8 +41,10 @@ import org.apache.flink.runtime.io.network.buffer.Buffer;
import org.apache.flink.runtime.io.network.buffer.Buffer.DataType;
import org.apache.flink.runtime.io.network.buffer.BufferBuilder;
import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
+import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
-import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
+import org.apache.flink.runtime.io.network.buffer.NoOpBufferPool;
import org.apache.flink.runtime.io.network.partition.InputChannelTestUtils;
import org.apache.flink.runtime.io.network.partition.PartitionNotFoundException;
import org.apache.flink.runtime.io.network.partition.PartitionProducerStateProvider;
@@ -474,8 +478,8 @@ class RemoteInputChannelTest {
@Test
void testAvailableBuffersLessThanRequiredBuffers() throws Exception {
// Setup
- final int numTotalBuffers = 16;
- final NetworkBufferPool networkBufferPool = new NetworkBufferPool(numTotalBuffers, 32);
+ final NetworkBufferPool networkBufferPool = new NetworkBufferPool(16, 32);
+ final int numFloatingBuffers = 14;
final SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
final RemoteInputChannel inputChannel = createRemoteInputChannel(inputGate);
@@ -485,7 +489,7 @@ class RemoteInputChannelTest {
final BufferPool bufferPool =
spy(
networkBufferPool.createBufferPool(
- numTotalBuffers, numTotalBuffers, numTotalBuffers));
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -643,8 +647,8 @@ class RemoteInputChannelTest {
@Test
void testAvailableBuffersEqualToRequiredBuffers() throws Exception {
// Setup
- final int numTotalBuffers = 16;
- final NetworkBufferPool networkBufferPool = new NetworkBufferPool(numTotalBuffers, 32);
+ final NetworkBufferPool networkBufferPool = new NetworkBufferPool(16, 32);
+ final int numFloatingBuffers = 14;
final SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
final RemoteInputChannel inputChannel = createRemoteInputChannel(inputGate);
@@ -654,7 +658,7 @@ class RemoteInputChannelTest {
final BufferPool bufferPool =
spy(
networkBufferPool.createBufferPool(
- numTotalBuffers, numTotalBuffers, numTotalBuffers));
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -732,8 +736,8 @@ class RemoteInputChannelTest {
@Test
void testAvailableBuffersMoreThanRequiredBuffers() throws Exception {
// Setup
- final int numTotalBuffers = 16;
- final NetworkBufferPool networkBufferPool = new NetworkBufferPool(numTotalBuffers, 32);
+ final NetworkBufferPool networkBufferPool = new NetworkBufferPool(16, 32);
+ final int numFloatingBuffers = 14;
final SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
final RemoteInputChannel inputChannel = createRemoteInputChannel(inputGate);
@@ -743,7 +747,7 @@ class RemoteInputChannelTest {
final BufferPool bufferPool =
spy(
networkBufferPool.createBufferPool(
- numTotalBuffers, numTotalBuffers, numTotalBuffers));
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -842,11 +846,9 @@ class RemoteInputChannelTest {
final int numExclusiveBuffers = 2;
final NetworkBufferPool networkBufferPool = new NetworkBufferPool(12, 32);
final int numFloatingBuffers = 3;
- final int numChannels = 3;
- final SingleInputGate inputGate = createSingleInputGate(numChannels, networkBufferPool);
- final int poolSize = numExclusiveBuffers * numChannels + numFloatingBuffers;
- final RemoteInputChannel[] inputChannels = new RemoteInputChannel[numChannels];
+ final SingleInputGate inputGate = createSingleInputGate(3, networkBufferPool);
+ final RemoteInputChannel[] inputChannels = new RemoteInputChannel[3];
inputChannels[0] = createRemoteInputChannel(inputGate);
inputChannels[1] = createRemoteInputChannel(inputGate);
inputChannels[2] = createRemoteInputChannel(inputGate);
@@ -854,7 +856,9 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- spy(networkBufferPool.createBufferPool(poolSize, poolSize, poolSize));
+ spy(
+ networkBufferPool.createBufferPool(
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputGate.requestPartitions();
@@ -1043,12 +1047,10 @@ class RemoteInputChannelTest {
void testConcurrentOnSenderBacklogAndRecycle() throws Exception {
// Setup
final int numExclusiveSegments = 120;
+ final NetworkBufferPool networkBufferPool = new NetworkBufferPool(248, 32);
final int numFloatingBuffers = 128;
- final int numTotalBuffers = numExclusiveSegments + numFloatingBuffers;
final int backlog = 128;
- final NetworkBufferPool networkBufferPool = new NetworkBufferPool(numTotalBuffers, 32);
-
final ExecutorService executor = Executors.newFixedThreadPool(3);
final SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
@@ -1059,7 +1061,7 @@ class RemoteInputChannelTest {
try {
final BufferPool bufferPool =
networkBufferPool.createBufferPool(
- numTotalBuffers, numTotalBuffers, numTotalBuffers);
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1109,9 +1111,8 @@ class RemoteInputChannelTest {
void testConcurrentRecycleAndRelease() throws Exception {
// Setup
final int numExclusiveSegments = 120;
+ final NetworkBufferPool networkBufferPool = new NetworkBufferPool(248, 32);
final int numFloatingBuffers = 128;
- final int numTotalBuffers = numExclusiveSegments + numFloatingBuffers;
- final NetworkBufferPool networkBufferPool = new NetworkBufferPool(numTotalBuffers, 32);
final ExecutorService executor = Executors.newFixedThreadPool(3);
@@ -1123,7 +1124,7 @@ class RemoteInputChannelTest {
try {
final BufferPool bufferPool =
networkBufferPool.createBufferPool(
- numTotalBuffers, numTotalBuffers, numTotalBuffers);
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1152,11 +1153,14 @@ class RemoteInputChannelTest {
.isZero();
assertThat(bufferPool.getNumberOfAvailableMemorySegments())
.withFailMessage(
- "There should be %d buffers available in local pool.", numTotalBuffers)
- .isEqualTo(numTotalBuffers);
+ "There should be %d buffers available in local pool.",
+ numFloatingBuffers)
+ .isEqualTo(numFloatingBuffers);
assertThat(networkBufferPool.getNumberOfAvailableMemorySegments())
- .withFailMessage("There should be no buffers available in global pool.")
- .isEqualTo(0);
+ .withFailMessage(
+ "There should be %d buffers available in global pool.",
+ numExclusiveSegments)
+ .isEqualTo(numExclusiveSegments);
} catch (Throwable t) {
thrown = t;
} finally {
@@ -1187,7 +1191,7 @@ class RemoteInputChannelTest {
try {
final BufferPool bufferPool =
networkBufferPool.createBufferPool(
- numTotalBuffers, numTotalBuffers, numTotalBuffers);
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1259,6 +1263,7 @@ class RemoteInputChannelTest {
@Test
void testConcurrentGetNextBufferAndRelease() throws Exception {
final int numTotalBuffers = 1_000;
+ final int numFloatingBuffers = 998;
final NetworkBufferPool networkBufferPool = new NetworkBufferPool(numTotalBuffers, 32);
final SingleInputGate inputGate = createSingleInputGate(1, networkBufferPool);
final RemoteInputChannel inputChannel = createRemoteInputChannel(inputGate);
@@ -1269,7 +1274,7 @@ class RemoteInputChannelTest {
try {
BufferPool bufferPool =
networkBufferPool.createBufferPool(
- numTotalBuffers, numTotalBuffers, numTotalBuffers);
+ numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1375,7 +1380,7 @@ class RemoteInputChannelTest {
@Test
void testOnUpstreamBlockedAndResumed() throws Exception {
- BufferPool bufferPool = new TestingBufferPool();
+ BufferPool bufferPool = new TestBufferPool();
SingleInputGate inputGate = createSingleInputGate(bufferPool);
RemoteInputChannel remoteChannel1 = createRemoteInputChannel(inputGate, 0, 2);
@@ -1424,7 +1429,7 @@ class RemoteInputChannelTest {
@Test
void testRequestBuffer() throws Exception {
- BufferPool bufferPool = new TestingBufferPool();
+ BufferPool bufferPool = new TestBufferPool();
SingleInputGate inputGate = createSingleInputGate(bufferPool);
RemoteInputChannel remoteChannel1 = createRemoteInputChannel(inputGate, 0, 2);
@@ -2081,4 +2086,13 @@ class RemoteInputChannelTest {
assertThat(delayMs).isEqualTo(expectedDelayMs);
}
}
+
+ private static final class TestBufferPool extends NoOpBufferPool {
+
+ @Override
+ public Buffer requestBuffer() {
+ MemorySegment segment = MemorySegmentFactory.allocateUnpooledSegment(1024);
+ return new NetworkBuffer(segment, FreeingBufferRecycler.INSTANCE);
+ }
+ }
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
index ab89c226c29..47111f8ae81 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
@@ -24,7 +24,7 @@ import org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter;
import org.apache.flink.runtime.io.network.NettyShuffleEnvironment;
import org.apache.flink.runtime.io.network.buffer.BufferDecompressor;
import org.apache.flink.runtime.io.network.buffer.BufferPool;
-import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
+import org.apache.flink.runtime.io.network.buffer.NoOpBufferPool;
import org.apache.flink.runtime.io.network.partition.InputChannelTestUtils;
import org.apache.flink.runtime.io.network.partition.PartitionProducerStateProvider;
import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
@@ -70,15 +70,12 @@ public class SingleInputGateBuilder {
private MemorySegmentProvider segmentProvider =
InputChannelTestUtils.StubMemorySegmentProvider.getInstance();
- private BufferPool bufferPool = null;
-
private ChannelStateWriter channelStateWriter = ChannelStateWriter.NO_OP;
@Nullable
private BiFunction<InputChannelBuilder, SingleInputGate, InputChannel> channelFactory = null;
- private SupplierWithException<BufferPool, IOException> bufferPoolFactory =
- TestingBufferPool::new;
+ private SupplierWithException<BufferPool, IOException> bufferPoolFactory = NoOpBufferPool::new;
private BufferDebloatConfiguration bufferDebloatConfiguration =
BufferDebloatConfiguration.fromConfiguration(new Configuration());
private Function<BufferDebloatConfiguration, ThroughputCalculator> createThroughputCalculator =
@@ -115,21 +112,13 @@ public class SingleInputGateBuilder {
}
public SingleInputGateBuilder setupBufferPoolFactory(NettyShuffleEnvironment environment) {
- return setupBufferPoolFactory(environment, 1);
- }
-
- public SingleInputGateBuilder setupBufferPoolFactory(
- NettyShuffleEnvironment environment, int minBuffers) {
NettyShuffleEnvironmentConfiguration config = environment.getConfiguration();
- return setupBufferPoolFactory(
- environment, minBuffers, config.floatingNetworkBuffersPerGate());
- }
-
- public SingleInputGateBuilder setupBufferPoolFactory(
- NettyShuffleEnvironment environment, int minBuffers, int maxBuffers) {
this.bufferPoolFactory =
SingleInputGateFactory.createBufferPoolFactory(
- environment.getNetworkBufferPool(), minBuffers, minBuffers, maxBuffers);
+ environment.getNetworkBufferPool(),
+ 1,
+ 1,
+ config.floatingNetworkBuffersPerGate());
this.segmentProvider = environment.getNetworkBufferPool();
return this;
}
@@ -149,11 +138,6 @@ public class SingleInputGateBuilder {
return this;
}
- public SingleInputGateBuilder setBufferPool(BufferPool bufferPool) {
- this.bufferPool = bufferPool;
- return this;
- }
-
/** Adds automatic initialization of all channels with the given factory. */
public SingleInputGateBuilder setChannelFactory(
BiFunction<InputChannelBuilder, SingleInputGate, InputChannel> channelFactory) {
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java
index 6d153b180d7..bb0b4ea5c38 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java
@@ -51,7 +51,7 @@ import org.apache.flink.runtime.io.network.buffer.BufferPool;
import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
-import org.apache.flink.runtime.io.network.buffer.TestingBufferPool;
+import org.apache.flink.runtime.io.network.buffer.NoOpBufferPool;
import org.apache.flink.runtime.io.network.partition.BufferAvailabilityListener;
import org.apache.flink.runtime.io.network.partition.BufferWritingResultPartition;
import org.apache.flink.runtime.io.network.partition.ChannelStateHolder;
@@ -204,7 +204,6 @@ class SingleInputGateTest extends InputGateTestBase {
void testPartitionRequestLogic() throws Exception {
final NettyShuffleEnvironment environment = new NettyShuffleEnvironmentBuilder().build();
final SingleInputGate gate = createInputGate(environment);
- gate.setup();
try (Closer closer = Closer.create()) {
closer.register(environment::close);
@@ -222,7 +221,7 @@ class SingleInputGateTest extends InputGateTestBase {
assertThat(remoteChannel).isInstanceOf(RemoteInputChannel.class);
assertThat(((RemoteInputChannel) remoteChannel).getPartitionRequestClient())
.isNotNull();
- assertThat(((RemoteInputChannel) remoteChannel).getNumExclusiveBuffers()).isEqualTo(2);
+ assertThat(((RemoteInputChannel) remoteChannel).getInitialCredit()).isEqualTo(2);
final InputChannel localChannel = gate.getChannel(1);
assertThat(localChannel).isInstanceOf(LocalInputChannel.class);
@@ -608,11 +607,7 @@ class SingleInputGateTest extends InputGateTestBase {
.setId(newPartitionId)
.setProducerLocation(ResourceID.generate())
.buildRemote();
- inputGate.setBufferPool(
- TestingBufferPool.builder()
- .setRequestMemorySegmentSupplier(
- () -> MemorySegmentFactory.allocateUnpooledSegment(bufferSize))
- .build());
+ inputGate.setBufferPool(new NoOpBufferPool());
inputGate.updateInputChannel(ResourceID.generate(), nettyShuffleDescriptor);
InputChannel newChannel = inputGate.getChannel(0);
@@ -702,7 +697,6 @@ class SingleInputGateTest extends InputGateTestBase {
SingleInputGate gate =
createSingleInputGate(partitionIds, ResultPartitionType.PIPELINED, netEnv);
gate.setChannelStateWriter(ChannelStateWriter.NO_OP);
- gate.setup();
gate.finishReadRecoveredState();
while (!gate.getStateConsumedFuture().isDone()) {
@@ -978,12 +972,7 @@ class SingleInputGateTest extends InputGateTestBase {
gate.setup();
assertThat(gate.getBufferPool()).isNotNull();
assertThat(gate.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(
- (gate.getNumberOfInputChannels()
- - gate
- .unsynchronizedGetNumberOfLocalInputChannels())
- * 2
- + 1);
+ .isEqualTo(gate.getInputChannels().size() * 2 + 1);
gate.finishReadRecoveredState();
while (!gate.getStateConsumedFuture().isDone()) {
@@ -1000,8 +989,7 @@ class SingleInputGateTest extends InputGateTestBase {
if (inputChannel instanceof RemoteInputChannel) {
assertThat(((RemoteInputChannel) inputChannel).getPartitionRequestClient())
.isNotNull();
- assertThat(((RemoteInputChannel) inputChannel).getNumExclusiveBuffers())
- .isEqualTo(2);
+ assertThat(((RemoteInputChannel) inputChannel).getInitialCredit()).isEqualTo(2);
} else if (inputChannel instanceof LocalInputChannel) {
assertThat(((LocalInputChannel) inputChannel).getSubpartitionView()).isNotNull();
}
@@ -1281,23 +1269,19 @@ class SingleInputGateTest extends InputGateTestBase {
for (InputChannel inputChannel : gate.inputChannels()) {
if (inputChannel instanceof RemoteInputChannel) {
- assertThat(((RemoteInputChannel) inputChannel).getNumExclusiveBuffers())
- .isEqualTo(0);
+ assertThat(((RemoteInputChannel) inputChannel).getInitialCredit()).isEqualTo(0);
}
}
- int numNonLocalInputChannels =
- gate.getNumberOfInputChannels()
- - gate.unsynchronizedGetNumberOfLocalInputChannels();
- int maxBuffersPerGate = 2 * numNonLocalInputChannels + 8;
- int expectedBuffersPerGate;
+ int maxBuffersPerGate = 2 * partitionIds.length * subpartitionRandSize + 8;
+ int minBuffersPerGate;
if (maxBuffersPerGate >= expectMaxRequiredBuffersPerGate.get()) {
- expectedBuffersPerGate = expectMaxRequiredBuffersPerGate.get();
+ minBuffersPerGate = expectMaxRequiredBuffersPerGate.get();
} else {
- expectedBuffersPerGate = 2 * numNonLocalInputChannels + 1;
+ minBuffersPerGate = 2 * partitionIds.length * subpartitionRandSize + 1;
}
assertThat(gate.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(expectedBuffersPerGate);
+ .isEqualTo(minBuffersPerGate);
assertThat(gate.getBufferPool().getMaxNumberOfMemorySegments())
.isEqualTo(maxBuffersPerGate);
}
diff --git a/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java b/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
index afd02301bb6..e2d66fcbbb4 100644
--- a/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
+++ b/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
@@ -64,8 +64,10 @@ public class SuccessAfterNetworkBuffersFailureITCase extends TestLogger {
private static Configuration getConfiguration() {
Configuration config = new Configuration();
config.set(TaskManagerOptions.MANAGED_MEMORY_SIZE, MemorySize.parse("20m"));
- config.set(TaskManagerOptions.NETWORK_MEMORY_MIN, MemorySize.ofMebiBytes(3L));
- config.set(TaskManagerOptions.NETWORK_MEMORY_MAX, MemorySize.ofMebiBytes(3L));
+ // Temporarily changes the memory to 4MB since the input gates will request more buffers
+ // than before, we will roll back it after the input gate side changes merged.
+ config.set(TaskManagerOptions.NETWORK_MEMORY_MIN, MemorySize.ofMebiBytes(4L));
+ config.set(TaskManagerOptions.NETWORK_MEMORY_MAX, MemorySize.ofMebiBytes(4L));
return config;
}
(flink) 06/06: Revert "[FLINK-33668][runtime] Redefine the redistribution logic of the NetworkBufferPool"
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
guoweijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 67d23fc45be45c02341a9cbd7dddea18ba40bca2
Author: Yuxin Tan <ta...@gmail.com>
AuthorDate: Fri Jun 21 14:18:22 2024 +0800
Revert "[FLINK-33668][runtime] Redefine the redistribution logic of the NetworkBufferPool"
This reverts commit 012b8936e119d1bfadadee3182cf86d25c6dd019.
---
.../runtime/io/network/buffer/BufferPool.java | 8 +-
.../io/network/buffer/BufferPoolFactory.java | 18 +-
.../runtime/io/network/buffer/LocalBufferPool.java | 90 ++++-----
.../io/network/buffer/NetworkBufferPool.java | 205 ++++++---------------
.../partition/BufferWritingResultPartition.java | 2 +-
.../network/partition/ResultPartitionFactory.java | 9 +-
.../partition/SortMergeResultPartition.java | 2 +-
.../partition/consumer/GateBuffersSpec.java | 34 ++--
.../partition/consumer/InputGateSpecUtils.java | 49 +++--
.../partition/consumer/SingleInputGateFactory.java | 16 +-
.../flink/runtime/shuffle/NettyShuffleUtils.java | 34 ++--
.../SequentialChannelStateReaderImplTest.java | 2 +-
.../io/network/NettyShuffleEnvironmentTest.java | 10 +-
.../api/writer/RecordWriterDelegateTest.java | 2 +-
.../io/network/api/writer/RecordWriterTest.java | 2 +-
.../io/network/buffer/BufferPoolFactoryTest.java | 186 +++++--------------
.../io/network/buffer/LocalBufferPoolTest.java | 37 +---
.../io/network/buffer/NetworkBufferPoolTest.java | 38 ++--
.../runtime/io/network/buffer/NoOpBufferPool.java | 7 +-
.../io/network/buffer/UnpooledBufferPool.java | 7 +-
...editBasedPartitionRequestClientHandlerTest.java | 12 +-
.../netty/NettyPartitionRequestClientTest.java | 8 +-
.../io/network/partition/DataBufferTest.java | 6 +-
.../io/network/partition/ResultPartitionTest.java | 6 +-
.../partition/SortMergeResultPartitionTest.java | 31 ++--
.../partition/consumer/GateBuffersSpecTest.java | 69 +++----
.../partition/consumer/LocalInputChannelTest.java | 4 +-
.../partition/consumer/RemoteInputChannelTest.java | 38 ++--
.../partition/consumer/SingleInputGateBuilder.java | 1 -
.../partition/consumer/SingleInputGateTest.java | 30 +--
.../partition/hybrid/HsMemoryDataManagerTest.java | 8 +-
.../partition/hybrid/HsResultPartitionTest.java | 26 +--
.../partition/hybrid/HsSubpartitionViewTest.java | 2 +-
.../tiered/shuffle/TieredResultPartitionTest.java | 21 +--
.../tiered/storage/HashBufferAccumulatorTest.java | 3 +-
.../tiered/storage/SortBufferAccumulatorTest.java | 3 +-
.../TieredStorageMemoryManagerImplTest.java | 13 +-
.../storage/TieredStorageSortBufferTest.java | 9 +-
.../runtime/shuffle/NettyShuffleUtilsTest.java | 12 +-
.../runtime/state/ChannelPersistenceITCase.java | 3 +-
.../AlignedCheckpointsMassiveRandomTest.java | 4 +-
.../checkpointing/CheckpointedInputGateTest.java | 6 +-
.../util/TestCheckpointedInputGateBuilder.java | 3 +-
.../SuccessAfterNetworkBuffersFailureITCase.java | 6 +-
44 files changed, 404 insertions(+), 678 deletions(-)
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPool.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPool.java
index f037eeabf20..c574607e28e 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPool.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPool.java
@@ -40,14 +40,8 @@ public interface BufferPool extends BufferProvider, BufferRecycler {
@Override
boolean isDestroyed();
- /**
- * Returns the number of expected memory segments of this buffer pool, representing a value that
- * adequately satisfies the requirements for buffer usage.
- */
- int getExpectedNumberOfMemorySegments();
-
/** Returns the number of guaranteed (minimum number of) memory segments of this buffer pool. */
- int getMinNumberOfMemorySegments();
+ int getNumberOfRequiredMemorySegments();
/**
* Returns the maximum number of memory segments this buffer pool should use.
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactory.java
index 261616cb8cc..04a0cf2d60a 100755
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactory.java
@@ -27,33 +27,27 @@ public interface BufferPoolFactory {
* Tries to create a buffer pool, which is guaranteed to provide at least the number of required
* buffers.
*
- * <p>The buffer pool is of dynamic size ranges from <tt>minUsedBuffers</tt> to
- * <tt>maxUsedBuffers</tt>, with <tt>numExpectedBuffers</tt> serving as the weight.
+ * <p>The buffer pool is of dynamic size with at least <tt>numRequiredBuffers</tt> buffers.
*
- * @param numExpectedBuffers the number of expected network buffers of this pool
- * @param minUsedBuffers minimum number of network buffers in this pool
+ * @param numRequiredBuffers minimum number of network buffers in this pool
* @param maxUsedBuffers maximum number of network buffers this pool offers
*/
- BufferPool createBufferPool(int numExpectedBuffers, int minUsedBuffers, int maxUsedBuffers)
- throws IOException;
+ BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers) throws IOException;
/**
* Tries to create a buffer pool with an owner, which is guaranteed to provide at least the
* number of required buffers.
*
- * <p>The buffer pool is of dynamic size ranges from <tt>minUsedBuffers</tt> to
- * <tt>maxUsedBuffers</tt>, with <tt>numExpectedBuffers</tt> serving as the weight.
+ * <p>The buffer pool is of dynamic size with at least <tt>numRequiredBuffers</tt> buffers.
*
- * @param numExpectedBuffers the number of expected network buffers of this pool
- * @param minUsedBuffers minimum number of network buffers in this pool
+ * @param numRequiredBuffers minimum number of network buffers in this pool
* @param maxUsedBuffers maximum number of network buffers this pool offers
* @param numSubpartitions number of subpartitions in this pool
* @param maxBuffersPerChannel maximum number of buffers to use for each channel
* @param maxOverdraftBuffersPerGate maximum number of overdraft buffers to use for each gate
*/
BufferPool createBufferPool(
- int numExpectedBuffers,
- int minUsedBuffers,
+ int numRequiredBuffers,
int maxUsedBuffers,
int numSubpartitions,
int maxBuffersPerChannel,
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
index c74d1a8e314..873414c6fe2 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java
@@ -76,6 +76,9 @@ public class LocalBufferPool implements BufferPool {
/** Global network buffer pool to get buffers from. */
private final NetworkBufferPool networkBufferPool;
+ /** The minimum number of required segments for this pool. */
+ private final int numberOfRequiredMemorySegments;
+
/**
* The currently available memory segments. These are segments, which have been requested from
* the network buffer pool and are currently not handed out as Buffer instances.
@@ -94,18 +97,6 @@ public class LocalBufferPool implements BufferPool {
*/
private final ArrayDeque<BufferListener> registeredListeners = new ArrayDeque<>();
- /**
- * The number of expected memory segments of this buffer pool.
- *
- * <p>Usually, the buffers in {@link NetworkBufferPool} do not exactly meet the expectations of
- * all {@link LocalBufferPool}s, so typically this value is used as a weight to allocate buffers
- * to each {@link LocalBufferPool}.
- */
- private final int expectedNumberOfMemorySegments;
-
- /** The number of guaranteed (minimum number of) memory segments of this buffer pool. */
- private final int minNumberOfMemorySegments;
-
/** Maximum number of network buffers to allocate. */
private final int maxNumberOfMemorySegments;
@@ -151,13 +142,12 @@ public class LocalBufferPool implements BufferPool {
* network buffers being available.
*
* @param networkBufferPool global network buffer pool to get buffers from
- * @param minNumberOfMemorySegments minimum number of network buffers
+ * @param numberOfRequiredMemorySegments minimum number of network buffers
*/
- LocalBufferPool(NetworkBufferPool networkBufferPool, int minNumberOfMemorySegments) {
+ LocalBufferPool(NetworkBufferPool networkBufferPool, int numberOfRequiredMemorySegments) {
this(
networkBufferPool,
- minNumberOfMemorySegments,
- minNumberOfMemorySegments,
+ numberOfRequiredMemorySegments,
Integer.MAX_VALUE,
0,
Integer.MAX_VALUE,
@@ -169,17 +159,16 @@ public class LocalBufferPool implements BufferPool {
* number of network buffers being available.
*
* @param networkBufferPool global network buffer pool to get buffers from
- * @param minNumberOfMemorySegments minimum number of network buffers
+ * @param numberOfRequiredMemorySegments minimum number of network buffers
* @param maxNumberOfMemorySegments maximum number of network buffers to allocate
*/
LocalBufferPool(
NetworkBufferPool networkBufferPool,
- int minNumberOfMemorySegments,
+ int numberOfRequiredMemorySegments,
int maxNumberOfMemorySegments) {
this(
networkBufferPool,
- minNumberOfMemorySegments,
- minNumberOfMemorySegments,
+ numberOfRequiredMemorySegments,
maxNumberOfMemorySegments,
0,
Integer.MAX_VALUE,
@@ -191,8 +180,7 @@ public class LocalBufferPool implements BufferPool {
* with a minimal and maximal number of network buffers being available.
*
* @param networkBufferPool global network buffer pool to get buffers from
- * @param expectedNumberOfMemorySegments expected number of network buffers
- * @param minNumberOfMemorySegments minimum number of network buffers
+ * @param numberOfRequiredMemorySegments minimum number of network buffers
* @param maxNumberOfMemorySegments maximum number of network buffers to allocate
* @param numberOfSubpartitions number of subpartitions
* @param maxBuffersPerChannel maximum number of buffers to use for each channel
@@ -200,39 +188,30 @@ public class LocalBufferPool implements BufferPool {
*/
LocalBufferPool(
NetworkBufferPool networkBufferPool,
- int expectedNumberOfMemorySegments,
- int minNumberOfMemorySegments,
+ int numberOfRequiredMemorySegments,
int maxNumberOfMemorySegments,
int numberOfSubpartitions,
int maxBuffersPerChannel,
int maxOverdraftBuffersPerGate) {
checkArgument(
- minNumberOfMemorySegments > 0,
- "Minimum number of memory segments (%s) should be larger than 0.",
- minNumberOfMemorySegments);
+ numberOfRequiredMemorySegments > 0,
+ "Required number of memory segments (%s) should be larger than 0.",
+ numberOfRequiredMemorySegments);
checkArgument(
- expectedNumberOfMemorySegments >= minNumberOfMemorySegments,
- "Minimum number of memory segments (%s) should not be larger than expected (%s).",
- minNumberOfMemorySegments,
- expectedNumberOfMemorySegments);
-
- checkArgument(
- maxNumberOfMemorySegments >= expectedNumberOfMemorySegments,
- "Maximum number of memory segments (%s) should not be smaller than expected (%s).",
+ maxNumberOfMemorySegments >= numberOfRequiredMemorySegments,
+ "Maximum number of memory segments (%s) should not be smaller than minimum (%s).",
maxNumberOfMemorySegments,
- expectedNumberOfMemorySegments);
+ numberOfRequiredMemorySegments);
LOG.debug(
- "Using a local buffer pool who has the minimum-expected-maximum number of buffers as {}-{}-{}.",
- minNumberOfMemorySegments,
- expectedNumberOfMemorySegments,
+ "Using a local buffer pool with {}-{} buffers",
+ numberOfRequiredMemorySegments,
maxNumberOfMemorySegments);
this.networkBufferPool = networkBufferPool;
- this.expectedNumberOfMemorySegments = expectedNumberOfMemorySegments;
- this.currentPoolSize = minNumberOfMemorySegments;
- this.minNumberOfMemorySegments = minNumberOfMemorySegments;
+ this.numberOfRequiredMemorySegments = numberOfRequiredMemorySegments;
+ this.currentPoolSize = numberOfRequiredMemorySegments;
this.maxNumberOfMemorySegments = maxNumberOfMemorySegments;
if (numberOfSubpartitions > 0) {
@@ -268,8 +247,8 @@ public class LocalBufferPool implements BufferPool {
@Override
public void reserveSegments(int numberOfSegmentsToReserve) throws IOException {
checkArgument(
- numberOfSegmentsToReserve <= minNumberOfMemorySegments,
- "Can not reserve more segments than number of minimum segments.");
+ numberOfSegmentsToReserve <= numberOfRequiredMemorySegments,
+ "Can not reserve more segments than number of required segments.");
CompletableFuture<?> toNotify = null;
synchronized (availableMemorySegments) {
@@ -293,13 +272,8 @@ public class LocalBufferPool implements BufferPool {
}
@Override
- public int getExpectedNumberOfMemorySegments() {
- return expectedNumberOfMemorySegments;
- }
-
- @Override
- public int getMinNumberOfMemorySegments() {
- return minNumberOfMemorySegments;
+ public int getNumberOfRequiredMemorySegments() {
+ return numberOfRequiredMemorySegments;
}
@Override
@@ -312,13 +286,13 @@ public class LocalBufferPool implements BufferPool {
*
* @return the same value as {@link #getMaxNumberOfMemorySegments()} for bounded pools. For
* unbounded pools it returns an approximation based upon {@link
- * #getExpectedNumberOfMemorySegments()}
+ * #getNumberOfRequiredMemorySegments()}
*/
public int getEstimatedNumberOfRequestedMemorySegments() {
if (maxNumberOfMemorySegments < NetworkBufferPool.UNBOUNDED_POOL_SIZE) {
return maxNumberOfMemorySegments;
} else {
- return getExpectedNumberOfMemorySegments();
+ return getNumberOfRequiredMemorySegments() * 2;
}
}
@@ -690,9 +664,9 @@ public class LocalBufferPool implements BufferPool {
CompletableFuture<?> toNotify;
synchronized (availableMemorySegments) {
checkArgument(
- numBuffers >= minNumberOfMemorySegments,
+ numBuffers >= numberOfRequiredMemorySegments,
"Buffer pool needs at least %s buffers, but tried to set to %s",
- minNumberOfMemorySegments,
+ numberOfRequiredMemorySegments,
numBuffers);
currentPoolSize = Math.min(numBuffers, maxNumberOfMemorySegments);
@@ -730,14 +704,12 @@ public class LocalBufferPool implements BufferPool {
public String toString() {
synchronized (availableMemorySegments) {
return String.format(
- this.hashCode()
- + "[size: %d, requested: %d, available: %d, expected: %d, min: %d, max: %d, listeners: %d,"
+ "[size: %d, required: %d, requested: %d, available: %d, max: %d, listeners: %d,"
+ "subpartitions: %d, maxBuffersPerChannel: %d, destroyed: %s]",
currentPoolSize,
+ numberOfRequiredMemorySegments,
numberOfRequestedMemorySegments,
availableMemorySegments.size(),
- expectedNumberOfMemorySegments,
- minNumberOfMemorySegments,
maxNumberOfMemorySegments,
registeredListeners.size(),
subpartitionBuffersCount.length,
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java
index 39d3697e7b9..ec5e77bf662 100755
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java
@@ -29,6 +29,7 @@ import org.apache.flink.core.memory.MemorySegmentProvider;
import org.apache.flink.runtime.io.AvailabilityProvider;
import org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel;
import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.MathUtils;
import org.apache.flink.util.Preconditions;
import org.slf4j.Logger;
@@ -42,10 +43,8 @@ import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
-import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
-import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
@@ -59,9 +58,7 @@ import static org.apache.flink.util.Preconditions.checkNotNull;
*
* <p>The NetworkBufferPool creates {@link LocalBufferPool}s from which the individual tasks draw
* the buffers for the network data transfer. When new local buffer pools are created, the
- * NetworkBufferPool dynamically redistributes the buffers between the pools. The redistribution
- * logic is to make the number of buffers of each buffer pool fall within the minimum and maximum
- * numbers while endeavoring to closely align with its individual expected number.
+ * NetworkBufferPool dynamically redistributes the buffers between the pools.
*/
public class NetworkBufferPool
implements BufferPoolFactory, MemorySegmentProvider, AvailabilityProvider {
@@ -453,24 +450,22 @@ public class NetworkBufferPool
// ------------------------------------------------------------------------
@Override
- public BufferPool createBufferPool(
- int numExpectedBuffers, int minUsedBuffers, int maxUsedBuffers) throws IOException {
+ public BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers)
+ throws IOException {
return internalCreateBufferPool(
- numExpectedBuffers, minUsedBuffers, maxUsedBuffers, 0, Integer.MAX_VALUE, 0);
+ numRequiredBuffers, maxUsedBuffers, 0, Integer.MAX_VALUE, 0);
}
@Override
public BufferPool createBufferPool(
- int numExpectedBuffers,
- int minUsedBuffers,
+ int numRequiredBuffers,
int maxUsedBuffers,
int numSubpartitions,
int maxBuffersPerChannel,
int maxOverdraftBuffersPerGate)
throws IOException {
return internalCreateBufferPool(
- numExpectedBuffers,
- minUsedBuffers,
+ numRequiredBuffers,
maxUsedBuffers,
numSubpartitions,
maxBuffersPerChannel,
@@ -478,8 +473,7 @@ public class NetworkBufferPool
}
private BufferPool internalCreateBufferPool(
- int numExpectedBuffers,
- int minUsedBuffers,
+ int numRequiredBuffers,
int maxUsedBuffers,
int numSubpartitions,
int maxBuffersPerChannel,
@@ -495,25 +489,24 @@ public class NetworkBufferPool
// Ensure that the number of required buffers can be satisfied.
// With dynamic memory management this should become obsolete.
- if (numTotalRequiredBuffers + minUsedBuffers > totalNumberOfMemorySegments) {
+ if (numTotalRequiredBuffers + numRequiredBuffers > totalNumberOfMemorySegments) {
throw new IOException(
String.format(
"Insufficient number of network buffers: "
+ "required %d, but only %d available. %s.",
- minUsedBuffers,
+ numRequiredBuffers,
totalNumberOfMemorySegments - numTotalRequiredBuffers,
getConfigDescription()));
}
- this.numTotalRequiredBuffers += minUsedBuffers;
+ this.numTotalRequiredBuffers += numRequiredBuffers;
// We are good to go, create a new buffer pool and redistribute
// non-fixed size buffers.
LocalBufferPool localBufferPool =
new LocalBufferPool(
this,
- numExpectedBuffers,
- minUsedBuffers,
+ numRequiredBuffers,
maxUsedBuffers,
numSubpartitions,
maxBuffersPerChannel,
@@ -521,7 +514,7 @@ public class NetworkBufferPool
allBufferPools.add(localBufferPool);
- if (minUsedBuffers < maxUsedBuffers) {
+ if (numRequiredBuffers < maxUsedBuffers) {
resizableBufferPools.add(localBufferPool);
}
@@ -539,7 +532,7 @@ public class NetworkBufferPool
synchronized (factoryLock) {
if (allBufferPools.remove(bufferPool)) {
- numTotalRequiredBuffers -= bufferPool.getMinNumberOfMemorySegments();
+ numTotalRequiredBuffers -= bufferPool.getNumberOfRequiredMemorySegments();
resizableBufferPools.remove(bufferPool);
redistributeBuffers();
@@ -549,7 +542,7 @@ public class NetworkBufferPool
/**
* Destroys all buffer pools that allocate their buffers from this buffer pool (created via
- * {@link #createBufferPool(int, int, int)}).
+ * {@link #createBufferPool(int, int)}).
*/
public void destroyAllBufferPools() {
synchronized (factoryLock) {
@@ -606,149 +599,73 @@ public class NetworkBufferPool
}
// All buffers, which are not among the required ones
- int numAvailableMemorySegment = totalNumberOfMemorySegments - numTotalRequiredBuffers;
+ final int numAvailableMemorySegment = totalNumberOfMemorySegments - numTotalRequiredBuffers;
if (numAvailableMemorySegment == 0) {
// in this case, we need to redistribute buffers so that every pool gets its minimum
for (LocalBufferPool bufferPool : resizableBufferPools) {
- bufferPool.setNumBuffers(bufferPool.getMinNumberOfMemorySegments());
+ bufferPool.setNumBuffers(bufferPool.getNumberOfRequiredMemorySegments());
}
return;
}
- // Calculates the number of buffers that can be redistributed and the total weight of buffer
- // pools that are resizable
- int totalWeight = 0;
- int numBuffersToBeRedistributed = numAvailableMemorySegment;
- for (LocalBufferPool bufferPool : resizableBufferPools) {
- numBuffersToBeRedistributed += bufferPool.getMinNumberOfMemorySegments();
- totalWeight += bufferPool.getExpectedNumberOfMemorySegments();
- }
-
- // First, all buffers are allocated proportionally according to the expected values of each
- // pool as weights. However, due to the constraints of minimum and maximum values, the
- // actual number of buffers distributed may be more or less than the total number of
- // buffers.
- int totalAllocated = 0;
- Map<LocalBufferPool, Integer> cachedPoolSize = new HashMap<>(resizableBufferPools.size());
- for (LocalBufferPool bufferPool : resizableBufferPools) {
- int expectedNumBuffers =
- bufferPool.getExpectedNumberOfMemorySegments()
- * numBuffersToBeRedistributed
- / totalWeight;
- int actualAllocated =
- Math.min(
- bufferPool.getMaxNumberOfMemorySegments(),
- Math.max(
- bufferPool.getMinNumberOfMemorySegments(), expectedNumBuffers));
- cachedPoolSize.put(bufferPool, actualAllocated);
- totalAllocated += actualAllocated;
- }
-
- // Now we need to deal with this difference, which may be greater than zero or less than
- // zero.
- int delta = numBuffersToBeRedistributed - totalAllocated;
+ /*
+ * With buffer pools being potentially limited, let's distribute the available memory
+ * segments based on the capacity of each buffer pool, i.e. the maximum number of segments
+ * an unlimited buffer pool can take is numAvailableMemorySegment, for limited buffer pools
+ * it may be less. Based on this and the sum of all these values (totalCapacity), we build
+ * a ratio that we use to distribute the buffers.
+ */
- int remaining = Integer.MAX_VALUE;
- while (remaining != 0) {
- remaining = redistributeBuffers(delta, cachedPoolSize);
+ long totalCapacity = 0; // long to avoid int overflow
- // Stop the loop iteration when there is no remaining segments to be redistributed
- // or all local buffer pools have reached the max number.
- if (remaining == delta) {
- break;
- }
- delta = remaining;
+ for (LocalBufferPool bufferPool : resizableBufferPools) {
+ int excessMax =
+ bufferPool.getMaxNumberOfMemorySegments()
+ - bufferPool.getNumberOfRequiredMemorySegments();
+ totalCapacity += Math.min(numAvailableMemorySegment, excessMax);
}
- for (LocalBufferPool bufferPool : resizableBufferPools) {
- bufferPool.setNumBuffers(
- cachedPoolSize.getOrDefault(
- bufferPool, bufferPool.getMinNumberOfMemorySegments()));
+ // no capacity to receive additional buffers?
+ if (totalCapacity == 0) {
+ return; // necessary to avoid div by zero when nothing to re-distribute
}
- }
- /**
- * @param delta the buffers to be redistributed.
- * @param cachedPoolSize the map to cache the intermediate result.
- * @return the remaining buffers that can continue to be redistributed.
- */
- private int redistributeBuffers(int delta, Map<LocalBufferPool, Integer> cachedPoolSize) {
- Set<LocalBufferPool> poolsToBeRedistributed = new HashSet<>();
+ // since one of the arguments of 'min(a,b)' is a positive int, this is actually
+ // guaranteed to be within the 'int' domain
+ // (we use a checked downCast to handle possible bugs more gracefully).
+ final int memorySegmentsToDistribute =
+ MathUtils.checkedDownCast(Math.min(numAvailableMemorySegment, totalCapacity));
- if (delta > 0) {
- // In this case, we need to allocate the remaining buffers to pools that have
- // not yet reached their maximum number.
+ long totalPartsUsed = 0; // of totalCapacity
+ int numDistributedMemorySegment = 0;
+ for (LocalBufferPool bufferPool : resizableBufferPools) {
+ int excessMax =
+ bufferPool.getMaxNumberOfMemorySegments()
+ - bufferPool.getNumberOfRequiredMemorySegments();
- int totalWeight = 0;
- for (LocalBufferPool bufferPool : resizableBufferPools) {
- if (cachedPoolSize.get(bufferPool) < bufferPool.getMaxNumberOfMemorySegments()) {
- poolsToBeRedistributed.add(bufferPool);
- totalWeight += bufferPool.getExpectedNumberOfMemorySegments();
- }
+ // shortcut
+ if (excessMax == 0) {
+ continue;
}
- int totalAllocated = 0;
- float pieceOfDelta = ((float) delta) / totalWeight;
- for (LocalBufferPool bufferPool : poolsToBeRedistributed) {
- int extraAllocated =
- Math.min(
- delta - totalAllocated,
- (int)
- Math.ceil(
- pieceOfDelta
- * bufferPool
- .getExpectedNumberOfMemorySegments()));
- int numBuffers =
- Math.min(
- bufferPool.getMaxNumberOfMemorySegments(),
- cachedPoolSize.get(bufferPool) + extraAllocated);
- totalAllocated += numBuffers - cachedPoolSize.get(bufferPool);
- cachedPoolSize.put(bufferPool, numBuffers);
- if (totalAllocated == delta) {
- break;
- }
- }
- return delta - totalAllocated;
- } else if (delta < 0) {
- // In this case, we need to take back the previously allocated buffers from pools that
- // have not yet reached their minimum number.
+ totalPartsUsed += Math.min(numAvailableMemorySegment, excessMax);
- int totalWeight = 0;
- for (LocalBufferPool bufferPool : resizableBufferPools) {
- if (cachedPoolSize.get(bufferPool) == bufferPool.getMinNumberOfMemorySegments()) {
- continue;
- }
- poolsToBeRedistributed.add(bufferPool);
- totalWeight += bufferPool.getExpectedNumberOfMemorySegments();
- }
+ // avoid remaining buffers by looking at the total capacity that should have been
+ // re-distributed up until here
+ // the downcast will always succeed, because both arguments of the subtraction are in
+ // the 'int' domain
+ final int mySize =
+ MathUtils.checkedDownCast(
+ memorySegmentsToDistribute * totalPartsUsed / totalCapacity
+ - numDistributedMemorySegment);
- int totalDeallocated = 0;
- int deltaAbs = Math.abs(delta);
- float pieceOfDelta = ((float) deltaAbs) / totalWeight;
- for (LocalBufferPool bufferPool : poolsToBeRedistributed) {
- int extraDeallocated =
- Math.min(
- deltaAbs - totalDeallocated,
- (int)
- Math.ceil(
- pieceOfDelta
- * bufferPool
- .getExpectedNumberOfMemorySegments()));
- int numBuffers =
- Math.max(
- bufferPool.getMinNumberOfMemorySegments(),
- cachedPoolSize.get(bufferPool) - extraDeallocated);
- totalDeallocated += cachedPoolSize.get(bufferPool) - numBuffers;
- cachedPoolSize.put(bufferPool, numBuffers);
- if (totalDeallocated == deltaAbs) {
- break;
- }
- }
- return delta + totalDeallocated;
- } else {
- return 0;
+ numDistributedMemorySegment += mySize;
+ bufferPool.setNumBuffers(bufferPool.getNumberOfRequiredMemorySegments() + mySize);
}
+
+ assert (totalPartsUsed == totalCapacity);
+ assert (numDistributedMemorySegment == memorySegmentsToDistribute);
}
private String getConfigDescription() {
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferWritingResultPartition.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferWritingResultPartition.java
index 70010f2b508..eae9260642a 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferWritingResultPartition.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/BufferWritingResultPartition.java
@@ -96,7 +96,7 @@ public abstract class BufferWritingResultPartition extends ResultPartition {
@Override
protected void setupInternal() throws IOException {
checkState(
- bufferPool.getMinNumberOfMemorySegments() >= getNumberOfSubpartitions(),
+ bufferPool.getNumberOfRequiredMemorySegments() >= getNumberOfSubpartitions(),
"Bug in result partition setup logic: Buffer pool has not enough guaranteed buffers for"
+ " this result partition.");
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
index 1bf21f8c7bc..a90b59270a1 100755
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
@@ -19,7 +19,6 @@
package org.apache.flink.runtime.io.network.partition;
import org.apache.flink.annotation.VisibleForTesting;
-import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.NettyShuffleEnvironmentOptions.CompressionCodec;
import org.apache.flink.runtime.deployment.ResultPartitionDeploymentDescriptor;
import org.apache.flink.runtime.io.disk.BatchShuffleReadBufferPool;
@@ -41,6 +40,7 @@ import org.apache.flink.util.FlinkRuntimeException;
import org.apache.flink.util.ProcessorArchitecture;
import org.apache.flink.util.function.SupplierWithException;
+import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -383,7 +383,7 @@ public class ResultPartitionFactory {
SupplierWithException<BufferPool, IOException> createBufferPoolFactory(
int numberOfSubpartitions, ResultPartitionType type) {
return () -> {
- Tuple3<Integer, Integer, Integer> tuple =
+ Pair<Integer, Integer> pair =
NettyShuffleUtils.getMinMaxNetworkBuffersPerResultPartition(
configuredNetworkBuffersPerChannel,
floatingNetworkBuffersPerGate,
@@ -397,9 +397,8 @@ public class ResultPartitionFactory {
type);
return bufferPoolFactory.createBufferPool(
- tuple.f0,
- tuple.f1,
- tuple.f2,
+ pair.getLeft(),
+ pair.getRight(),
numberOfSubpartitions,
maxBuffersPerChannel,
isOverdraftBufferNeeded(type) ? maxOverdraftBuffersPerGate : 0);
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
index 4d31c9ebd9b..5873d4c59fa 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
@@ -331,7 +331,7 @@ public class SortMergeResultPartition extends ResultPartition {
}
private void requestGuaranteedBuffers() throws IOException {
- int numRequiredBuffer = bufferPool.getMinNumberOfMemorySegments();
+ int numRequiredBuffer = bufferPool.getNumberOfRequiredMemorySegments();
if (numRequiredBuffer < 2) {
throw new IOException(
String.format(
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpec.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpec.java
index b223240c277..0579414c7c6 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpec.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpec.java
@@ -26,36 +26,36 @@ public class GateBuffersSpec {
private final int effectiveExclusiveBuffersPerChannel;
- private final int expectedBuffersPerGate;
+ private final int requiredFloatingBuffers;
- private final int minBuffersPerGate;
+ private final int totalFloatingBuffers;
- private final int maxBuffersPerGate;
+ private final int targetTotalBuffersPerGate;
GateBuffersSpec(
int effectiveExclusiveBuffersPerChannel,
- int expectedBuffersPerGate,
- int minBuffersPerGate,
- int maxBuffersPerGate) {
+ int requiredFloatingBuffers,
+ int totalFloatingBuffers,
+ int targetTotalBuffersPerGate) {
this.effectiveExclusiveBuffersPerChannel = effectiveExclusiveBuffersPerChannel;
- this.expectedBuffersPerGate = expectedBuffersPerGate;
- this.minBuffersPerGate = minBuffersPerGate;
- this.maxBuffersPerGate = maxBuffersPerGate;
+ this.requiredFloatingBuffers = requiredFloatingBuffers;
+ this.totalFloatingBuffers = totalFloatingBuffers;
+ this.targetTotalBuffersPerGate = targetTotalBuffersPerGate;
}
- int getExpectedBuffersPerGate() {
- return expectedBuffersPerGate;
+ int getRequiredFloatingBuffers() {
+ return requiredFloatingBuffers;
}
- int getMinBuffersPerGate() {
- return minBuffersPerGate;
- }
-
- public int getMaxBuffersPerGate() {
- return maxBuffersPerGate;
+ int getTotalFloatingBuffers() {
+ return totalFloatingBuffers;
}
int getEffectiveExclusiveBuffersPerChannel() {
return effectiveExclusiveBuffersPerChannel;
}
+
+ public int targetTotalBuffersPerGate() {
+ return targetTotalBuffersPerGate;
+ }
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
index f415ff6165f..0093a5d61df 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputGateSpecUtils.java
@@ -20,11 +20,11 @@ package org.apache.flink.runtime.io.network.partition.consumer;
import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
-import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils;
import java.util.Optional;
import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
/** Utils to manage the specs of the {@link InputGate}, for example, {@link GateBuffersSpec}. */
public class InputGateSpecUtils {
@@ -43,32 +43,39 @@ public class InputGateSpecUtils {
int maxRequiredBuffersThresholdPerGate =
getEffectiveMaxRequiredBuffersPerGate(
partitionType, configuredMaxRequiredBuffersPerGate, enableTieredStorage);
- int targetExpectedBuffersPerGate =
- getExpectedBuffersTargetPerGate(
+ int targetRequiredBuffersPerGate =
+ getRequiredBuffersTargetPerGate(
numInputChannels, configuredNetworkBuffersPerChannel);
- int maxBuffersPerGate =
- getMaxBuffersPerGate(
+ int targetTotalBuffersPerGate =
+ getTotalBuffersTargetPerGate(
numInputChannels,
configuredNetworkBuffersPerChannel,
configuredFloatingNetworkBuffersPerGate);
- int expectedBuffersPerGate =
- Math.min(maxRequiredBuffersThresholdPerGate, targetExpectedBuffersPerGate);
+ int requiredBuffersPerGate =
+ Math.min(maxRequiredBuffersThresholdPerGate, targetRequiredBuffersPerGate);
+
int effectiveExclusiveBuffersPerChannel =
getExclusiveBuffersPerChannel(
configuredNetworkBuffersPerChannel,
numInputChannels,
- expectedBuffersPerGate);
- int minBuffersPerGate =
- partitionType.isHybridResultPartition() && enableTieredStorage
- ? TieredStorageUtils.getMinBuffersPerGate()
- : expectedBuffersPerGate;
- expectedBuffersPerGate = Math.max(minBuffersPerGate, expectedBuffersPerGate);
+ requiredBuffersPerGate);
+ int effectiveExclusiveBuffersPerGate =
+ getEffectiveExclusiveBuffersPerGate(
+ numInputChannels, effectiveExclusiveBuffersPerChannel);
+
+ int requiredFloatingBuffers = requiredBuffersPerGate - effectiveExclusiveBuffersPerGate;
+ int totalFloatingBuffers = targetTotalBuffersPerGate - effectiveExclusiveBuffersPerGate;
+
+ checkState(requiredFloatingBuffers > 0, "Must be positive.");
+ checkState(
+ requiredFloatingBuffers <= totalFloatingBuffers,
+ "Wrong number of floating buffers.");
return new GateBuffersSpec(
effectiveExclusiveBuffersPerChannel,
- expectedBuffersPerGate,
- minBuffersPerGate,
- maxBuffersPerGate);
+ requiredFloatingBuffers,
+ totalFloatingBuffers,
+ targetTotalBuffersPerGate);
}
@VisibleForTesting
@@ -105,17 +112,21 @@ public class InputGateSpecUtils {
(requiredBuffersPerGate - 1) / numInputChannels);
}
- private static int getExpectedBuffersTargetPerGate(
+ private static int getRequiredBuffersTargetPerGate(
int numInputChannels, int configuredNetworkBuffersPerChannel) {
return numInputChannels * configuredNetworkBuffersPerChannel + 1;
}
- /** */
- private static int getMaxBuffersPerGate(
+ private static int getTotalBuffersTargetPerGate(
int numInputChannels,
int configuredNetworkBuffersPerChannel,
int configuredFloatingBuffersPerGate) {
return numInputChannels * configuredNetworkBuffersPerChannel
+ configuredFloatingBuffersPerGate;
}
+
+ private static int getEffectiveExclusiveBuffersPerGate(
+ int numInputChannels, int effectiveExclusiveBuffersPerChannel) {
+ return effectiveExclusiveBuffersPerChannel * numInputChannels;
+ }
}
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
index 17669ac81f6..8e36a53eb44 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateFactory.java
@@ -57,6 +57,7 @@ import org.apache.flink.util.MathUtils;
import org.apache.flink.util.clock.SystemClock;
import org.apache.flink.util.function.SupplierWithException;
+import org.apache.commons.lang3.tuple.Pair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -171,9 +172,8 @@ public class SingleInputGateFactory {
SupplierWithException<BufferPool, IOException> bufferPoolFactory =
createBufferPoolFactory(
networkBufferPool,
- gateBuffersSpec.getExpectedBuffersPerGate(),
- gateBuffersSpec.getMinBuffersPerGate(),
- gateBuffersSpec.getMaxBuffersPerGate());
+ gateBuffersSpec.getRequiredFloatingBuffers(),
+ gateBuffersSpec.getTotalFloatingBuffers());
BufferDecompressor bufferDecompressor = null;
if (igdd.getConsumedPartitionType().supportCompression()
@@ -446,12 +446,10 @@ public class SingleInputGateFactory {
@VisibleForTesting
static SupplierWithException<BufferPool, IOException> createBufferPoolFactory(
BufferPoolFactory bufferPoolFactory,
- int expectedBuffersPerGate,
- int minBuffersPerGate,
- int maxBuffersPerGate) {
- return () ->
- bufferPoolFactory.createBufferPool(
- expectedBuffersPerGate, minBuffersPerGate, maxBuffersPerGate);
+ int minFloatingBuffersPerGate,
+ int maxFloatingBuffersPerGate) {
+ Pair<Integer, Integer> pair = Pair.of(minFloatingBuffersPerGate, maxFloatingBuffersPerGate);
+ return () -> bufferPoolFactory.createBufferPool(pair.getLeft(), pair.getRight());
}
/** Statistics of input channels. */
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java b/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java
index 73c54ce1ff4..2606e6d72ec 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleUtils.java
@@ -18,7 +18,6 @@
package org.apache.flink.runtime.shuffle;
-import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool;
import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
import org.apache.flink.runtime.io.network.partition.consumer.GateBuffersSpec;
@@ -40,10 +39,6 @@ import static org.apache.flink.util.Preconditions.checkState;
*/
public class NettyShuffleUtils {
- // Temporarily declare the default value here, it would be moved to the configuration class
- // later.
- private static final int DEFAULT_HYBRID_SHUFFLE_MIN_BUFFERS = 8;
-
/**
* Calculates and returns the number of required exclusive network buffers per input channel.
*/
@@ -63,10 +58,10 @@ public class NettyShuffleUtils {
}
/**
- * Calculates and returns local network buffer pool size used by the result partition. The value
- * in the returned tuple respectively represent the expected, min and max buffers of the pool.
+ * Calculates and returns local network buffer pool size used by the result partition. The
+ * left/right value of the returned pair represent the min/max buffers require by the pool.
*/
- public static Tuple3<Integer, Integer, Integer> getMinMaxNetworkBuffersPerResultPartition(
+ public static Pair<Integer, Integer> getMinMaxNetworkBuffersPerResultPartition(
final int configuredNetworkBuffersPerChannel,
final int numFloatingBuffersPerGate,
final int sortShuffleMinParallelism,
@@ -78,35 +73,28 @@ public class NettyShuffleUtils {
boolean isSortShuffle =
type.isBlockingOrBlockingPersistentResultPartition()
&& numSubpartitions >= sortShuffleMinParallelism;
- int expected;
+ int min;
if (isSortShuffle) {
- expected = sortShuffleMinBuffers;
+ min = sortShuffleMinBuffers;
} else {
- expected =
+ min =
enableTieredStorage
? Math.min(tieredStoreExclusiveBuffers, numSubpartitions + 1)
: (numSubpartitions + 1);
}
-
- int min = expected;
- if (type.isHybridResultPartition()) {
- min = Math.min(DEFAULT_HYBRID_SHUFFLE_MIN_BUFFERS, expected);
- }
-
int max =
type.isBounded()
? numSubpartitions * configuredNetworkBuffersPerChannel
+ numFloatingBuffersPerGate
: (isSortShuffle
- ? 4 * numSubpartitions
+ ? Math.max(min, 4 * numSubpartitions)
: NetworkBufferPool.UNBOUNDED_POOL_SIZE);
- max = Math.max(max, expected);
// for each upstream hash-based blocking/pipelined subpartition, at least one buffer is
// needed even the configured network buffers per channel is 0 and this behavior is for
// performance. If it's not guaranteed that each subpartition can get at least one buffer,
// more partial buffers with little data will be outputted to network/disk and recycled to
// be used by other subpartitions which can not get a buffer for data caching.
- return Tuple3.of(expected, min, max);
+ return Pair.of(min, Math.max(min, max));
}
public static int computeNetworkBuffersForAnnouncing(
@@ -171,7 +159,7 @@ public class NettyShuffleUtils {
type,
numInputChannels,
false);
- return gateBuffersSpec.getMaxBuffersPerGate();
+ return gateBuffersSpec.targetTotalBuffersPerGate();
}
private static int getNumBuffersToAnnounceForResultPartition(
@@ -182,7 +170,7 @@ public class NettyShuffleUtils {
int sortShuffleMinBuffers,
int numSubpartitions) {
- Tuple3<Integer, Integer, Integer> tuple =
+ Pair<Integer, Integer> minAndMax =
getMinMaxNetworkBuffersPerResultPartition(
configuredNetworkBuffersPerChannel,
floatingBuffersPerGate,
@@ -200,7 +188,7 @@ public class NettyShuffleUtils {
// should take the max value;
// 2. For blocking shuffle, it is back pressure free and floating buffers can be recycled
// in time, so that the minimum required buffers would be enough.
- int ret = type.canBePipelinedConsumed() ? tuple.f2 : tuple.f0;
+ int ret = type.canBePipelinedConsumed() ? minAndMax.getRight() : minAndMax.getLeft();
if (ret == Integer.MAX_VALUE) {
// Should never reach this branch. Result partition will allocate an unbounded
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java
index b7d7c192c9c..568d9b22810 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/channel/SequentialChannelStateReaderImplTest.java
@@ -211,7 +211,7 @@ public class SequentialChannelStateReaderImplTest {
.setSingleInputGateIndex(i)
.setBufferPoolFactory(
networkBufferPool.createBufferPool(
- 1, 1, buffersPerChannel))
+ 1, buffersPerChannel))
.setSegmentProvider(networkBufferPool)
.setChannelFactory(
(builder, gate) ->
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java
index 3d18d22eee4..ae792398de4 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentTest.java
@@ -320,7 +320,7 @@ class NettyShuffleEnvironmentTest {
.isEqualTo(expectedRp4Buffers);
for (ResultPartition rp : resultPartitions) {
- assertThat(rp.getBufferPool().getExpectedNumberOfMemorySegments())
+ assertThat(rp.getBufferPool().getNumberOfRequiredMemorySegments())
.isEqualTo(rp.getNumberOfSubpartitions() + 1);
assertThat(rp.getBufferPool().getNumBuffers())
.isEqualTo(rp.getNumberOfSubpartitions() + 1);
@@ -328,10 +328,10 @@ class NettyShuffleEnvironmentTest {
// verify buffer pools for the input gates (NOTE: credit-based uses minimum required buffers
// for exclusive buffers not managed by the buffer pool)
- assertThat(ig1.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
- assertThat(ig2.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
- assertThat(ig3.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
- assertThat(ig4.getBufferPool().getExpectedNumberOfMemorySegments()).isOne();
+ assertThat(ig1.getBufferPool().getNumberOfRequiredMemorySegments()).isOne();
+ assertThat(ig2.getBufferPool().getNumberOfRequiredMemorySegments()).isOne();
+ assertThat(ig3.getBufferPool().getNumberOfRequiredMemorySegments()).isOne();
+ assertThat(ig4.getBufferPool().getNumberOfRequiredMemorySegments()).isOne();
assertThat(ig1.getBufferPool().getMaxNumberOfMemorySegments()).isEqualTo(floatingBuffers);
assertThat(ig2.getBufferPool().getMaxNumberOfMemorySegments()).isEqualTo(floatingBuffers);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterDelegateTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterDelegateTest.java
index e7cbee016f5..ba7579a2456 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterDelegateTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterDelegateTest.java
@@ -125,7 +125,7 @@ class RecordWriterDelegateTest {
}
private static RecordWriter createRecordWriter(NetworkBufferPool globalPool) throws Exception {
- final BufferPool localPool = globalPool.createBufferPool(1, 1, 1, 1, Integer.MAX_VALUE, 0);
+ final BufferPool localPool = globalPool.createBufferPool(1, 1, 1, Integer.MAX_VALUE, 0);
final ResultPartitionWriter partition =
new ResultPartitionBuilder().setBufferPoolFactory(() -> localPool).build();
partition.setup();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java
index b7778d14fb2..22903144e8d 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java
@@ -313,7 +313,7 @@ class RecordWriterTest {
void testIsAvailableOrNot() throws Exception {
// setup
final NetworkBufferPool globalPool = new NetworkBufferPool(10, 128);
- final BufferPool localPool = globalPool.createBufferPool(1, 1, 1, 1, Integer.MAX_VALUE, 0);
+ final BufferPool localPool = globalPool.createBufferPool(1, 1, 1, Integer.MAX_VALUE, 0);
final ResultPartitionWriter resultPartition =
new ResultPartitionBuilder().setBufferPoolFactory(() -> localPool).build();
resultPartition.setup();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java
index 77237193320..e60c2806fa7 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/BufferPoolFactoryTest.java
@@ -18,7 +18,6 @@
package org.apache.flink.runtime.io.network.buffer;
-import org.apache.flink.core.fs.CloseableRegistry;
import org.apache.flink.core.memory.MemorySegment;
import org.junit.jupiter.api.AfterEach;
@@ -71,7 +70,6 @@ class BufferPoolFactoryTest {
assertThatThrownBy(
() ->
networkBufferPool.createBufferPool(
- networkBufferPool.getTotalNumberOfMemorySegments() + 1,
networkBufferPool.getTotalNumberOfMemorySegments() + 1,
Integer.MAX_VALUE))
.isInstanceOf(IOException.class)
@@ -83,16 +81,13 @@ class BufferPoolFactoryTest {
void testRequireMoreThanPossible2() throws IOException {
BufferPool bufferPool = null;
try {
- bufferPool =
- networkBufferPool.createBufferPool(
- numBuffers / 2 + 1, numBuffers / 2 + 1, numBuffers);
+ bufferPool = networkBufferPool.createBufferPool(numBuffers / 2 + 1, numBuffers);
assertThatThrownBy(
() ->
networkBufferPool.createBufferPool(
- numBuffers / 2 + 1, numBuffers / 2 + 1, numBuffers))
+ numBuffers / 2 + 1, numBuffers))
.isInstanceOf(IOException.class)
.hasMessageContaining("Insufficient number of network buffers");
-
} finally {
if (bufferPool != null) {
bufferPool.lazyDestroy();
@@ -115,9 +110,7 @@ class BufferPoolFactoryTest {
List<Buffer> buffers = new ArrayList<>(numBuffers);
BufferPool bufferPool1 = null, bufferPool2 = null;
try {
- bufferPool1 =
- networkBufferPool.createBufferPool(
- buffersToTakeFromPool2, buffersToTakeFromPool2, numBuffers);
+ bufferPool1 = networkBufferPool.createBufferPool(buffersToTakeFromPool2, numBuffers);
// take more buffers than the minimum required
for (int i = 0; i < buffersToTakeFromPool1; ++i) {
@@ -130,14 +123,12 @@ class BufferPoolFactoryTest {
assertThat(bufferPool1.getNumBuffers()).isEqualTo(numBuffers);
// create a second pool which requires more buffers than are available at the moment
- bufferPool2 =
- networkBufferPool.createBufferPool(
- buffersToTakeFromPool1, buffersToTakeFromPool1, numBuffers);
+ bufferPool2 = networkBufferPool.createBufferPool(buffersToTakeFromPool1, numBuffers);
assertThat(bufferPool2.getNumBuffers())
- .isEqualTo(bufferPool2.getExpectedNumberOfMemorySegments());
+ .isEqualTo(bufferPool2.getNumberOfRequiredMemorySegments());
assertThat(bufferPool1.getNumBuffers())
- .isEqualTo(bufferPool1.getExpectedNumberOfMemorySegments());
+ .isEqualTo(bufferPool1.getNumberOfRequiredMemorySegments());
assertThat(bufferPool1.requestBuffer()).isNull();
// take all remaining buffers
@@ -183,10 +174,10 @@ class BufferPoolFactoryTest {
@Test
void testBoundedPools() throws IOException {
- BufferPool bufferPool1 = networkBufferPool.createBufferPool(1, 1, 1);
+ BufferPool bufferPool1 = networkBufferPool.createBufferPool(1, 1);
assertThat(bufferPool1.getNumBuffers()).isOne();
- BufferPool bufferPool2 = networkBufferPool.createBufferPool(1, 1, 2);
+ BufferPool bufferPool2 = networkBufferPool.createBufferPool(1, 2);
assertThat(bufferPool2.getNumBuffers()).isEqualTo(2);
bufferPool1.lazyDestroy();
@@ -195,7 +186,7 @@ class BufferPoolFactoryTest {
@Test
void testSingleManagedPoolGetsAll() throws IOException {
- BufferPool bufferPool = networkBufferPool.createBufferPool(1, 1, Integer.MAX_VALUE);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
assertThat(bufferPool.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments());
@@ -205,9 +196,9 @@ class BufferPoolFactoryTest {
@Test
void testSingleManagedPoolGetsAllExceptFixedOnes() throws IOException {
- BufferPool fixedBufferPool = networkBufferPool.createBufferPool(24, 24, 24);
+ BufferPool fixedBufferPool = networkBufferPool.createBufferPool(24, 24);
- BufferPool flexibleBufferPool = networkBufferPool.createBufferPool(1, 1, Integer.MAX_VALUE);
+ BufferPool flexibleBufferPool = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
assertThat(fixedBufferPool.getNumBuffers()).isEqualTo(24);
assertThat(flexibleBufferPool.getNumBuffers())
@@ -221,11 +212,11 @@ class BufferPoolFactoryTest {
@Test
void testUniformDistribution() throws IOException {
- BufferPool first = networkBufferPool.createBufferPool(1, 1, Integer.MAX_VALUE);
+ BufferPool first = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments());
- BufferPool second = networkBufferPool.createBufferPool(1, 1, Integer.MAX_VALUE);
+ BufferPool second = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments() / 2);
assertThat(second.getNumBuffers())
@@ -243,17 +234,13 @@ class BufferPoolFactoryTest {
void testUniformDistributionAllBuffers() throws IOException {
BufferPool first =
networkBufferPool.createBufferPool(
- networkBufferPool.getTotalNumberOfMemorySegments() / 2,
- networkBufferPool.getTotalNumberOfMemorySegments() / 2,
- Integer.MAX_VALUE);
+ networkBufferPool.getTotalNumberOfMemorySegments() / 2, Integer.MAX_VALUE);
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments());
BufferPool second =
networkBufferPool.createBufferPool(
- networkBufferPool.getTotalNumberOfMemorySegments() / 2,
- networkBufferPool.getTotalNumberOfMemorySegments() / 2,
- Integer.MAX_VALUE);
+ networkBufferPool.getTotalNumberOfMemorySegments() / 2, Integer.MAX_VALUE);
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments() / 2);
assertThat(second.getNumBuffers())
@@ -267,13 +254,13 @@ class BufferPoolFactoryTest {
void testUniformDistributionBounded1() throws IOException {
BufferPool first =
networkBufferPool.createBufferPool(
- 1, 1, networkBufferPool.getTotalNumberOfMemorySegments());
+ 1, networkBufferPool.getTotalNumberOfMemorySegments());
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments());
BufferPool second =
networkBufferPool.createBufferPool(
- 1, 1, networkBufferPool.getTotalNumberOfMemorySegments());
+ 1, networkBufferPool.getTotalNumberOfMemorySegments());
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments() / 2);
assertThat(second.getNumBuffers())
@@ -285,10 +272,10 @@ class BufferPoolFactoryTest {
@Test
void testUniformDistributionBounded2() throws IOException {
- BufferPool first = networkBufferPool.createBufferPool(1, 1, 10);
+ BufferPool first = networkBufferPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isEqualTo(10);
- BufferPool second = networkBufferPool.createBufferPool(1, 1, 10);
+ BufferPool second = networkBufferPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isEqualTo(10);
assertThat(second.getNumBuffers()).isEqualTo(10);
@@ -300,16 +287,16 @@ class BufferPoolFactoryTest {
void testUniformDistributionBounded3() throws IOException {
NetworkBufferPool globalPool = new NetworkBufferPool(3, 128);
try {
- BufferPool first = globalPool.createBufferPool(1, 1, 10);
+ BufferPool first = globalPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isEqualTo(3);
- BufferPool second = globalPool.createBufferPool(1, 1, 10);
+ BufferPool second = globalPool.createBufferPool(1, 10);
// the order of which buffer pool received 2 or 1 buffer is undefined
assertThat(first.getNumBuffers() + second.getNumBuffers()).isEqualTo(3);
assertThat(first.getNumBuffers()).isNotEqualTo(3);
assertThat(second.getNumBuffers()).isNotEqualTo(3);
- BufferPool third = globalPool.createBufferPool(1, 1, 10);
+ BufferPool third = globalPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isOne();
assertThat(second.getNumBuffers()).isOne();
assertThat(third.getNumBuffers()).isOne();
@@ -335,14 +322,14 @@ class BufferPoolFactoryTest {
void testUniformDistributionBounded4() throws IOException {
NetworkBufferPool globalPool = new NetworkBufferPool(10, 128);
try {
- BufferPool first = globalPool.createBufferPool(1, 1, 10);
+ BufferPool first = globalPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isEqualTo(10);
List<MemorySegment> segmentList1 = globalPool.requestUnpooledMemorySegments(2);
assertThat(segmentList1).hasSize(2);
assertThat(first.getNumBuffers()).isEqualTo(8);
- BufferPool second = globalPool.createBufferPool(1, 1, 10);
+ BufferPool second = globalPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isEqualTo(4);
assertThat(second.getNumBuffers()).isEqualTo(4);
@@ -398,37 +385,39 @@ class BufferPoolFactoryTest {
void testBufferRedistributionMixed1() throws IOException {
// try multiple times for various orders during redistribution
for (int i = 0; i < 1_000; ++i) {
- BufferPool first = networkBufferPool.createBufferPool(10, 1, 10);
+ BufferPool first = networkBufferPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isEqualTo(10);
- BufferPool second = networkBufferPool.createBufferPool(10, 1, 10);
+ BufferPool second = networkBufferPool.createBufferPool(1, 10);
assertThat(first.getNumBuffers()).isEqualTo(10);
assertThat(second.getNumBuffers()).isEqualTo(10);
- BufferPool third = networkBufferPool.createBufferPool(10, 1, Integer.MAX_VALUE);
+ BufferPool third = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
// note: exact buffer distribution depends on the order during the redistribution
- for (BufferPool bp : new BufferPool[] {first, second}) {
+ for (BufferPool bp : new BufferPool[] {first, second, third}) {
+ final int avail = numBuffers - 3;
+ int size =
+ avail
+ * Math.min(avail, bp.getMaxNumberOfMemorySegments() - 1)
+ / (avail + 20 - 2)
+ + 1;
assertThat(bp.getNumBuffers())
.withFailMessage("Wrong buffer pool size after redistribution")
- .isEqualTo(10);
+ .isIn(size, size + 1);
}
- assertThat(third.getNumBuffers())
- .withFailMessage("Wrong buffer pool size after redistribution")
- .isEqualTo(numBuffers - 10 * 2);
-
- BufferPool fourth = networkBufferPool.createBufferPool(10, 1, Integer.MAX_VALUE);
+ BufferPool fourth = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
// note: exact buffer distribution depends on the order during the redistribution
- for (BufferPool bp : new BufferPool[] {first, second}) {
- assertThat(bp.getNumBuffers())
- .withFailMessage("Wrong buffer pool size after redistribution")
- .isEqualTo(10);
- }
-
- for (BufferPool bp : new BufferPool[] {third, fourth}) {
+ for (BufferPool bp : new BufferPool[] {first, second, third, fourth}) {
+ final int avail = numBuffers - 4;
+ int size =
+ avail
+ * Math.min(avail, bp.getMaxNumberOfMemorySegments() - 1)
+ / (2 * avail + 20 - 2)
+ + 1;
assertThat(bp.getNumBuffers())
.withFailMessage("Wrong buffer pool size after redistribution")
- .isEqualTo((numBuffers - 10 * 2) / 2);
+ .isIn(size, size + 1);
}
Stream.of(first, second, third, fourth).forEach(BufferPool::lazyDestroy);
@@ -437,85 +426,6 @@ class BufferPoolFactoryTest {
}
}
- @Test
- void testRedistributeBuffersByExpected() throws IOException {
- final int numBuffers = 32;
- try (CloseableRegistry closeableRegistry = new CloseableRegistry()) {
- NetworkBufferPool globalPool = new NetworkBufferPool(numBuffers, 128);
- closeableRegistry.registerCloseable(globalPool::destroy);
-
- BufferPool pool1 = globalPool.createBufferPool(4, 2, Integer.MAX_VALUE);
- BufferPool pool2 = globalPool.createBufferPool(5, 2, Integer.MAX_VALUE);
- BufferPool pool3 = globalPool.createBufferPool(7, 2, Integer.MAX_VALUE);
-
- assertThat(pool1.getNumBuffers()).isEqualTo(8);
- assertThat(pool2.getNumBuffers()).isEqualTo(10);
- assertThat(pool3.getNumBuffers()).isEqualTo(14);
-
- Stream.of(pool1, pool2, pool3).forEach(BufferPool::lazyDestroy);
- }
- }
-
- @Test
- void testRedistributeNothing() throws IOException {
- final int numBuffers = 4;
- try (CloseableRegistry closeableRegistry = new CloseableRegistry()) {
- NetworkBufferPool globalPool = new NetworkBufferPool(numBuffers, 128);
- closeableRegistry.registerCloseable(globalPool::destroy);
-
- BufferPool pool1 = globalPool.createBufferPool(10, 2, Integer.MAX_VALUE);
- BufferPool pool2 = globalPool.createBufferPool(10, 2, Integer.MAX_VALUE);
-
- assertThat(pool1.getNumBuffers()).isEqualTo(2);
- assertThat(pool2.getNumBuffers()).isEqualTo(2);
-
- Stream.of(pool1, pool2).forEach(BufferPool::lazyDestroy);
- }
- }
-
- @Test
- void testProportionNotMetDueToMinUsed() throws IOException {
- final int numBuffers = 32;
- int pieceOfBuffers = numBuffers / 4;
- try (CloseableRegistry closeableRegistry = new CloseableRegistry()) {
- NetworkBufferPool globalPool = new NetworkBufferPool(numBuffers, 128);
- closeableRegistry.registerCloseable(globalPool::destroy);
-
- BufferPool pool1 = globalPool.createBufferPool(pieceOfBuffers * 3 / 2, 1, numBuffers);
- BufferPool pool2 =
- globalPool.createBufferPool(
- pieceOfBuffers * 3 / 2, pieceOfBuffers + 3, numBuffers);
- BufferPool pool3 = globalPool.createBufferPool(pieceOfBuffers * 3, 1, numBuffers);
-
- assertThat(pool1.getNumBuffers()).isEqualTo(pieceOfBuffers - 1);
- assertThat(pool2.getNumBuffers()).isEqualTo(pieceOfBuffers + 3);
- assertThat(pool3.getNumBuffers()).isEqualTo(pieceOfBuffers * 2 - 2);
-
- Stream.of(pool1, pool2, pool3).forEach(BufferPool::lazyDestroy);
- }
- }
-
- @Test
- void testProportionNotMetDueToMaxUsed() throws IOException {
- final int numBuffers = 32;
- int pieceOfBuffers = numBuffers / 4;
- try (CloseableRegistry closeableRegistry = new CloseableRegistry()) {
- NetworkBufferPool globalPool = new NetworkBufferPool(numBuffers, 128);
- closeableRegistry.registerCloseable(globalPool::destroy);
-
- BufferPool pool1 = globalPool.createBufferPool(pieceOfBuffers / 2, 1, numBuffers);
- BufferPool pool2 =
- globalPool.createBufferPool(pieceOfBuffers / 2, 1, pieceOfBuffers - 3);
- BufferPool pool3 = globalPool.createBufferPool(pieceOfBuffers, 1, numBuffers);
-
- assertThat(pool1.getNumBuffers()).isEqualTo(pieceOfBuffers + 1);
- assertThat(pool2.getNumBuffers()).isEqualTo(pieceOfBuffers - 3);
- assertThat(pool3.getNumBuffers()).isEqualTo(pieceOfBuffers * 2 + 2);
-
- Stream.of(pool1, pool2, pool3).forEach(BufferPool::lazyDestroy);
- }
- }
-
@Test
void testAllDistributed() throws IOException {
// try multiple times for various orders during redistribution
@@ -533,9 +443,7 @@ class BufferPoolFactoryTest {
random.nextBoolean()
? Integer.MAX_VALUE
: Math.max(1, random.nextInt(10) + numRequiredBuffers);
- pools.add(
- networkBufferPool.createBufferPool(
- numRequiredBuffers, numRequiredBuffers, maxUsedBuffers));
+ pools.add(networkBufferPool.createBufferPool(numRequiredBuffers, maxUsedBuffers));
maxTotalUsed = Math.min(numBuffers, maxTotalUsed + maxUsedBuffers);
// after every iteration, all buffers (up to maxTotalUsed) must be distributed
@@ -554,12 +462,12 @@ class BufferPoolFactoryTest {
@Test
void testCreateDestroy() throws IOException {
- BufferPool first = networkBufferPool.createBufferPool(1, 1, Integer.MAX_VALUE);
+ BufferPool first = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments());
- BufferPool second = networkBufferPool.createBufferPool(1, 1, Integer.MAX_VALUE);
+ BufferPool second = networkBufferPool.createBufferPool(1, Integer.MAX_VALUE);
assertThat(first.getNumBuffers())
.isEqualTo(networkBufferPool.getTotalNumberOfMemorySegments() / 2);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPoolTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPoolTest.java
index 7d4e3bf7dd9..6f407b03a08 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPoolTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPoolTest.java
@@ -90,27 +90,12 @@ class LocalBufferPoolTest {
networkBufferPool.destroy();
}
- @Test
- void testCreateIllegalBufferPool() {
- NetworkBufferPool networkBufferPool =
- new NetworkBufferPool(2, memorySegmentSize, Duration.ofSeconds(2));
-
- assertThatThrownBy(() -> new LocalBufferPool(networkBufferPool, 0, 2, 3, 2, 2, 2))
- .isInstanceOf(IllegalArgumentException.class);
-
- assertThatThrownBy(() -> new LocalBufferPool(networkBufferPool, 1, 2, 3, 2, 2, 2))
- .isInstanceOf(IllegalArgumentException.class);
-
- assertThatThrownBy(() -> new LocalBufferPool(networkBufferPool, 4, 2, 3, 2, 2, 2))
- .isInstanceOf(IllegalArgumentException.class);
- }
-
@Test
void testReserveSegments() throws Exception {
NetworkBufferPool networkBufferPool =
new NetworkBufferPool(2, memorySegmentSize, Duration.ofSeconds(2));
try {
- BufferPool bufferPool1 = networkBufferPool.createBufferPool(1, 1, 2);
+ BufferPool bufferPool1 = networkBufferPool.createBufferPool(1, 2);
assertThatThrownBy(() -> bufferPool1.reserveSegments(2))
.isInstanceOf(IllegalArgumentException.class);
@@ -120,7 +105,7 @@ class LocalBufferPoolTest {
buffers.add(bufferPool1.requestBuffer());
assertThat(buffers).hasSize(2);
- BufferPool bufferPool2 = networkBufferPool.createBufferPool(1, 1, 10);
+ BufferPool bufferPool2 = networkBufferPool.createBufferPool(1, 10);
assertThatThrownBy(() -> bufferPool2.reserveSegments(1))
.isInstanceOf(IOException.class);
assertThat(bufferPool2.isAvailable()).isFalse();
@@ -129,7 +114,7 @@ class LocalBufferPoolTest {
bufferPool1.lazyDestroy();
bufferPool2.lazyDestroy();
- BufferPool bufferPool3 = networkBufferPool.createBufferPool(2, 2, 10);
+ BufferPool bufferPool3 = networkBufferPool.createBufferPool(2, 10);
assertThat(bufferPool3.getNumberOfAvailableMemorySegments()).isOne();
bufferPool3.reserveSegments(2);
assertThat(bufferPool3.getNumberOfAvailableMemorySegments()).isEqualTo(2);
@@ -149,8 +134,7 @@ class LocalBufferPoolTest {
int segmentsToReserve = 2;
NetworkBufferPool globalPool = new NetworkBufferPool(totalSegments, memorySegmentSize);
- BufferPool localPool1 =
- globalPool.createBufferPool(segmentsToReserve, segmentsToReserve, totalSegments);
+ BufferPool localPool1 = globalPool.createBufferPool(segmentsToReserve, totalSegments);
List<MemorySegment> segments = new ArrayList<>();
try {
@@ -158,9 +142,7 @@ class LocalBufferPoolTest {
segments.add(localPool1.requestMemorySegmentBlocking());
}
- BufferPool localPool2 =
- globalPool.createBufferPool(
- segmentsToReserve, segmentsToReserve, totalSegments);
+ BufferPool localPool2 = globalPool.createBufferPool(segmentsToReserve, totalSegments);
// the segment reserve thread will be blocked for no buffer is available
Thread reserveThread =
new Thread(
@@ -306,7 +288,6 @@ class LocalBufferPoolTest {
new LocalBufferPool(
networkBufferPool,
requiredMemorySegments,
- requiredMemorySegments,
maxMemorySegments,
0,
Integer.MAX_VALUE,
@@ -393,7 +374,6 @@ class LocalBufferPoolTest {
new LocalBufferPool(
networkBufferPool,
requiredMemorySegments,
- requiredMemorySegments,
maxMemorySegments,
0,
Integer.MAX_VALUE,
@@ -448,14 +428,14 @@ class LocalBufferPoolTest {
private void testRequestBuffersOnRecycle(boolean supportOverdraftBuffer) throws Exception {
BufferPool bufferPool1 =
networkBufferPool.createBufferPool(
- 512, 512, 2048, 0, Integer.MAX_VALUE, supportOverdraftBuffer ? 5 : 0);
+ 512, 2048, 0, Integer.MAX_VALUE, supportOverdraftBuffer ? 5 : 0);
List<MemorySegment> segments = new ArrayList<>();
for (int i = 0; i < 1023; i++) {
segments.add(bufferPool1.requestMemorySegmentBlocking());
}
BufferPool bufferPool2 =
networkBufferPool.createBufferPool(
- 512, 512, 512, 0, Integer.MAX_VALUE, supportOverdraftBuffer ? 5 : 0);
+ 512, 512, 0, Integer.MAX_VALUE, supportOverdraftBuffer ? 5 : 0);
List<MemorySegment> segments2 = new ArrayList<>();
CheckedThread checkedThread =
new CheckedThread() {
@@ -680,7 +660,7 @@ class LocalBufferPoolTest {
@Test
void testMaxBuffersPerChannelAndAvailability() throws Exception {
localBufferPool.lazyDestroy();
- localBufferPool = new LocalBufferPool(networkBufferPool, 1, 1, Integer.MAX_VALUE, 3, 2, 0);
+ localBufferPool = new LocalBufferPool(networkBufferPool, 1, Integer.MAX_VALUE, 3, 2, 0);
localBufferPool.setNumBuffers(10);
assertThat(localBufferPool.getAvailableFuture()).isDone();
@@ -820,7 +800,6 @@ class LocalBufferPoolTest {
new LocalBufferPool(
networkBufferPool,
1,
- 1,
Integer.MAX_VALUE,
numberOfChannels,
maxBuffersPerChannel,
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java
index ffdc399320b..6c3731716fa 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPoolTest.java
@@ -59,13 +59,13 @@ class NetworkBufferPoolTest {
assertThat(globalPool.isDestroyed()).isTrue();
- assertThatThrownBy(() -> globalPool.createBufferPool(2, 2, 2))
+ assertThatThrownBy(() -> globalPool.createBufferPool(2, 2))
.isInstanceOf(IllegalStateException.class);
- assertThatThrownBy(() -> globalPool.createBufferPool(2, 2, 10))
+ assertThatThrownBy(() -> globalPool.createBufferPool(2, 10))
.isInstanceOf(IllegalStateException.class);
- assertThatThrownBy(() -> globalPool.createBufferPool(2, 2, Integer.MAX_VALUE))
+ assertThatThrownBy(() -> globalPool.createBufferPool(2, Integer.MAX_VALUE))
.isInstanceOf(IllegalStateException.class);
}
@@ -126,13 +126,13 @@ class NetworkBufferPoolTest {
void testDestroyAll() throws IOException {
NetworkBufferPool globalPool = new NetworkBufferPool(10, 128);
- BufferPool fixedPool = globalPool.createBufferPool(2, 2, 2);
- BufferPool boundedPool = globalPool.createBufferPool(1, 1, 1);
- BufferPool nonFixedPool = globalPool.createBufferPool(5, 5, Integer.MAX_VALUE);
+ BufferPool fixedPool = globalPool.createBufferPool(2, 2);
+ BufferPool boundedPool = globalPool.createBufferPool(1, 1);
+ BufferPool nonFixedPool = globalPool.createBufferPool(5, Integer.MAX_VALUE);
- assertThat(fixedPool.getExpectedNumberOfMemorySegments()).isEqualTo(2);
- assertThat(boundedPool.getExpectedNumberOfMemorySegments()).isOne();
- assertThat(nonFixedPool.getExpectedNumberOfMemorySegments()).isEqualTo(5);
+ assertThat(fixedPool.getNumberOfRequiredMemorySegments()).isEqualTo(2);
+ assertThat(boundedPool.getNumberOfRequiredMemorySegments()).isOne();
+ assertThat(nonFixedPool.getNumberOfRequiredMemorySegments()).isEqualTo(5);
// actually, the buffer pool sizes may be different due to rounding and based on the
// internal order of
@@ -188,7 +188,7 @@ class NetworkBufferPoolTest {
assertThatThrownBy(nonFixedPool::requestBuffer).isInstanceOf(CancelTaskException.class);
// can create a new pool now
- assertThat(globalPool.createBufferPool(10, 10, Integer.MAX_VALUE)).isNotNull();
+ assertThat(globalPool.createBufferPool(10, Integer.MAX_VALUE)).isNotNull();
}
/**
@@ -288,21 +288,20 @@ class NetworkBufferPoolTest {
NetworkBufferPool globalPool = new NetworkBufferPool(50, 128);
closeableRegistry.registerCloseable(globalPool::destroy);
- BufferPool bufferPool1 = globalPool.createBufferPool(10, 10, 20);
+ BufferPool bufferPool1 = globalPool.createBufferPool(10, 20);
assertThat(globalPool.getEstimatedNumberOfRequestedMemorySegments()).isEqualTo(20);
assertThat(globalPool.getEstimatedRequestedSegmentsUsage()).isEqualTo(40);
assertThat(globalPool.getUsageWarning()).isEmpty();
closeableRegistry.registerCloseable(
- (globalPool.createBufferPool(10, 5, Integer.MAX_VALUE))::lazyDestroy);
+ (globalPool.createBufferPool(5, Integer.MAX_VALUE))::lazyDestroy);
assertThat(globalPool.getEstimatedNumberOfRequestedMemorySegments()).isEqualTo(30);
assertThat(globalPool.getEstimatedRequestedSegmentsUsage()).isEqualTo(60);
assertThat(globalPool.getUsageWarning()).isEmpty();
- closeableRegistry.registerCloseable(
- (globalPool.createBufferPool(10, 10, 30))::lazyDestroy);
+ closeableRegistry.registerCloseable((globalPool.createBufferPool(10, 30))::lazyDestroy);
assertThat(globalPool.getEstimatedNumberOfRequestedMemorySegments()).isEqualTo(60);
assertThat(globalPool.getEstimatedRequestedSegmentsUsage()).isEqualTo(120);
@@ -317,7 +316,7 @@ class NetworkBufferPoolTest {
+ "missingMemory=1.250kb (1280 bytes))");
assertThat(globalPool.getUsageWarning()).isEmpty();
- BufferPool bufferPool2 = globalPool.createBufferPool(10, 10, 20);
+ BufferPool bufferPool2 = globalPool.createBufferPool(10, 20);
assertThat(globalPool.getEstimatedNumberOfRequestedMemorySegments()).isEqualTo(80);
assertThat(globalPool.getEstimatedRequestedSegmentsUsage()).isEqualTo(160);
@@ -372,7 +371,7 @@ class NetworkBufferPoolTest {
Thread bufferRecycler = null;
BufferPool lbp1 = null;
try {
- lbp1 = networkBufferPool.createBufferPool(numBuffers / 2, numBuffers / 2, numBuffers);
+ lbp1 = networkBufferPool.createBufferPool(numBuffers / 2, numBuffers);
// take all buffers (more than the minimum required)
for (int i = 0; i < numBuffers; ++i) {
@@ -495,7 +494,7 @@ class NetworkBufferPoolTest {
// test indirectly for NetworkBufferPool#numTotalRequiredBuffers being correct:
// -> creating a new buffer pool should not fail
- globalPool.createBufferPool(10, 10, 10);
+ globalPool.createBufferPool(10, 10);
} finally {
globalPool.destroy();
}
@@ -514,7 +513,7 @@ class NetworkBufferPoolTest {
NetworkBufferPool globalPool =
new NetworkBufferPool(numBuffers, 128, requestSegmentsTimeout);
- BufferPool localBufferPool = globalPool.createBufferPool(1, 1, numBuffers);
+ BufferPool localBufferPool = globalPool.createBufferPool(1, numBuffers);
for (int i = 0; i < numBuffers; ++i) {
localBufferPool.requestBuffer();
}
@@ -653,8 +652,7 @@ class NetworkBufferPoolTest {
// create local buffer pools
for (int i = 0; i < numLocalBufferPool; ++i) {
final BufferPool localPool =
- globalPool.createBufferPool(
- localPoolRequiredSize, localPoolRequiredSize, localPoolMaxSize);
+ globalPool.createBufferPool(localPoolRequiredSize, localPoolMaxSize);
localBufferPools.add(localPool);
assertThat(localPool.getAvailableFuture()).isDone();
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NoOpBufferPool.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NoOpBufferPool.java
index bb5189215c7..d4ab33fa053 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NoOpBufferPool.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/NoOpBufferPool.java
@@ -80,12 +80,7 @@ public class NoOpBufferPool implements BufferPool {
}
@Override
- public int getExpectedNumberOfMemorySegments() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getMinNumberOfMemorySegments() {
+ public int getNumberOfRequiredMemorySegments() {
throw new UnsupportedOperationException();
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/UnpooledBufferPool.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/UnpooledBufferPool.java
index 265ae77bb3b..0ff07afe5f2 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/UnpooledBufferPool.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/buffer/UnpooledBufferPool.java
@@ -81,12 +81,7 @@ public class UnpooledBufferPool implements BufferPool {
}
@Override
- public int getExpectedNumberOfMemorySegments() {
- return Integer.MAX_VALUE;
- }
-
- @Override
- public int getMinNumberOfMemorySegments() {
+ public int getNumberOfRequiredMemorySegments() {
return Integer.MAX_VALUE;
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
index ef83f0ce7f3..307e5d9dcb0 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/CreditBasedPartitionRequestClientHandlerTest.java
@@ -175,7 +175,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
InputChannelBuilder.newBuilder().buildRemoteChannel(inputGate);
try {
inputGate.setInputChannels(inputChannel);
- final BufferPool bufferPool = networkBufferPool.createBufferPool(8, 8, 8);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(8, 8);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -221,7 +221,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
inputGate.setInputChannels(inputChannel);
try {
- BufferPool bufferPool = networkBufferPool.createBufferPool(8, 8, 8);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(8, 8);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -262,7 +262,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
inputGate.setInputChannels(inputChannel);
try {
- BufferPool bufferPool = networkBufferPool.createBufferPool(8, 8, 8);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(8, 8);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -402,7 +402,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
inputChannels[1] = createRemoteInputChannel(inputGate, client);
try {
inputGate.setInputChannels(inputChannels);
- final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -525,7 +525,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
final RemoteInputChannel inputChannel = createRemoteInputChannel(inputGate, client);
try {
inputGate.setInputChannels(inputChannel);
- final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -688,7 +688,7 @@ class CreditBasedPartitionRequestClientHandlerTest {
inputChannels[1] = createRemoteInputChannel(inputGate, client);
try {
inputGate.setInputChannels(inputChannels);
- final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClientTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClientTest.java
index 4170774ede2..a45b5434174 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClientTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyPartitionRequestClientTest.java
@@ -110,7 +110,7 @@ class NettyPartitionRequestClientTest {
try {
inputGate.setInputChannels(inputChannel);
- final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
@@ -171,7 +171,7 @@ class NettyPartitionRequestClientTest {
try {
inputGate.setInputChannels(inputChannel);
- final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -207,7 +207,7 @@ class NettyPartitionRequestClientTest {
final RemoteInputChannel inputChannel = createRemoteInputChannel(inputGate, client);
try {
- final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
+ final BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -245,7 +245,7 @@ class NettyPartitionRequestClientTest {
RemoteInputChannel inputChannel = createRemoteInputChannel(inputGate, client);
try {
- BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6, 6);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(6, 6);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/DataBufferTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/DataBufferTest.java
index 0ce7906cc7a..3f62f4fe377 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/DataBufferTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/DataBufferTest.java
@@ -402,8 +402,7 @@ class DataBufferTest {
int recordSize = (bufferPoolSize - 1) * bufferSize;
NetworkBufferPool globalPool = new NetworkBufferPool(bufferPoolSize, bufferSize);
- BufferPool bufferPool =
- globalPool.createBufferPool(bufferPoolSize, bufferPoolSize, bufferPoolSize);
+ BufferPool bufferPool = globalPool.createBufferPool(bufferPoolSize, bufferPoolSize);
LinkedList<MemorySegment> segments = new LinkedList<>();
for (int i = 0; i < bufferPoolSize; ++i) {
@@ -435,8 +434,7 @@ class DataBufferTest {
int bufferPoolSize, int bufferSize, int numSubpartitions, int[] customReadOrder)
throws Exception {
NetworkBufferPool globalPool = new NetworkBufferPool(bufferPoolSize, bufferSize);
- BufferPool bufferPool =
- globalPool.createBufferPool(bufferPoolSize, bufferPoolSize, bufferPoolSize);
+ BufferPool bufferPool = globalPool.createBufferPool(bufferPoolSize, bufferPoolSize);
LinkedList<MemorySegment> segments = new LinkedList<>();
for (int i = 0; i < bufferPoolSize; ++i) {
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java
index 8e7b79f92dd..8132f64897d 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionTest.java
@@ -333,7 +333,7 @@ class ResultPartitionTest {
partition.setup();
BufferPool bufferPool = partition.getBufferPool();
// verify the amount of buffers in created local pool
- assertThat(bufferPool.getExpectedNumberOfMemorySegments())
+ assertThat(bufferPool.getNumberOfRequiredMemorySegments())
.isEqualTo(partition.getNumberOfSubpartitions() + 1);
if (type.isBounded()) {
final int maxNumBuffers =
@@ -369,7 +369,7 @@ class ResultPartitionTest {
// setup
int bufferSize = 1024;
NetworkBufferPool globalPool = new NetworkBufferPool(10, bufferSize);
- BufferPool localPool = globalPool.createBufferPool(1, 1, 1, 1, Integer.MAX_VALUE, 0);
+ BufferPool localPool = globalPool.createBufferPool(1, 1, 1, Integer.MAX_VALUE, 0);
BufferWritingResultPartition resultPartition =
(BufferWritingResultPartition)
new ResultPartitionBuilder().setBufferPoolFactory(() -> localPool).build();
@@ -489,7 +489,7 @@ class ResultPartitionTest {
int maxBufferSize = 2 * recordSize;
// create a pool with just 1 buffer - so that the test times out in case of back-pressure
NetworkBufferPool globalPool = new NetworkBufferPool(1, maxBufferSize);
- BufferPool localPool = globalPool.createBufferPool(1, 1, 1, 1, Integer.MAX_VALUE, 0);
+ BufferPool localPool = globalPool.createBufferPool(1, 1, 1, Integer.MAX_VALUE, 0);
ResultPartition resultPartition =
new ResultPartitionBuilder().setBufferPoolFactory(() -> localPool).build();
resultPartition.setup();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionTest.java
index b50215ed3df..18c812e8d6e 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionTest.java
@@ -116,7 +116,7 @@ class SortMergeResultPartitionTest {
int numRecords = 1000;
Random random = new Random();
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
SortMergeResultPartition partition =
createSortMergedPartition(numSubpartitions, bufferPool);
@@ -250,7 +250,7 @@ class SortMergeResultPartitionTest {
@TestTemplate
void testWriteLargeRecord() throws Exception {
int numBuffers = useHashDataBuffer ? 100 : 15;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
SortMergeResultPartition partition = createSortMergedPartition(10, bufferPool);
ByteBuffer recordWritten = generateRandomData(bufferSize * numBuffers, new Random());
@@ -291,7 +291,7 @@ class SortMergeResultPartitionTest {
int numBuffers = useHashDataBuffer ? 100 : 15;
int numRecords = 10000;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
SortMergeResultPartition partition =
createSortMergedPartition(numSubpartitions, bufferPool);
@@ -326,7 +326,7 @@ class SortMergeResultPartitionTest {
void testReleaseWhileWriting() throws Exception {
int numBuffers = useHashDataBuffer ? 100 : 15;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
SortMergeResultPartition partition = createSortMergedPartition(10, bufferPool);
assertThat(bufferPool.bestEffortGetNumOfUsedBuffers()).isEqualTo(numBuffers);
@@ -349,7 +349,7 @@ class SortMergeResultPartitionTest {
void testRelease() throws Exception {
int numBuffers = useHashDataBuffer ? 100 : 15;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
SortMergeResultPartition partition = createSortMergedPartition(10, bufferPool);
assertThat(bufferPool.bestEffortGetNumOfUsedBuffers()).isEqualTo(numBuffers);
@@ -383,7 +383,7 @@ class SortMergeResultPartitionTest {
void testCloseReleasesAllBuffers() throws Exception {
int numBuffers = useHashDataBuffer ? 100 : 15;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
SortMergeResultPartition partition = createSortMergedPartition(10, bufferPool);
assertThat(bufferPool.bestEffortGetNumOfUsedBuffers()).isEqualTo(numBuffers);
@@ -398,7 +398,7 @@ class SortMergeResultPartitionTest {
@TestTemplate
void testReadUnfinishedPartition() throws Exception {
- BufferPool bufferPool = globalPool.createBufferPool(10, 10, 10);
+ BufferPool bufferPool = globalPool.createBufferPool(10, 10);
SortMergeResultPartition partition = createSortMergedPartition(10, bufferPool);
assertThatThrownBy(
() ->
@@ -410,7 +410,7 @@ class SortMergeResultPartitionTest {
@TestTemplate
void testReadReleasedPartition() throws Exception {
- BufferPool bufferPool = globalPool.createBufferPool(10, 10, 10);
+ BufferPool bufferPool = globalPool.createBufferPool(10, 10);
SortMergeResultPartition partition = createSortMergedPartition(10, bufferPool);
partition.finish();
partition.release();
@@ -437,7 +437,7 @@ class SortMergeResultPartitionTest {
void testNetworkBufferReservation() throws IOException {
int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, 2 * numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, 2 * numBuffers);
SortMergeResultPartition partition = createSortMergedPartition(1, bufferPool);
assertThat(bufferPool.bestEffortGetNumOfUsedBuffers()).isEqualTo(numBuffers);
@@ -455,8 +455,7 @@ class SortMergeResultPartitionTest {
BatchShuffleReadBufferPool.NUM_BYTES_PER_REQUEST, bufferSize);
BufferPool bufferPool =
- networkBufferPool.createBufferPool(
- numNetworkBuffers, numNetworkBuffers, numNetworkBuffers);
+ networkBufferPool.createBufferPool(numNetworkBuffers, numNetworkBuffers);
SortMergeResultPartition partition =
createSortMergedPartition(1, bufferPool, readBufferPool);
for (int i = 0; i < numNetworkBuffers; ++i) {
@@ -476,9 +475,7 @@ class SortMergeResultPartitionTest {
new ResultSubpartitionIndexSet(0), listener);
BufferPool bufferPool1 =
networkBufferPool.createBufferPool(
- numNetworkBuffers / 2,
- numNetworkBuffers / 2,
- numNetworkBuffers);
+ numNetworkBuffers / 2, numNetworkBuffers);
SortMergeResultPartition partition1 =
createSortMergedPartition(1, bufferPool1);
readAndEmitData(view, partition1);
@@ -498,9 +495,7 @@ class SortMergeResultPartitionTest {
condition1.await();
BufferPool bufferPool2 =
networkBufferPool.createBufferPool(
- numNetworkBuffers / 2,
- numNetworkBuffers / 2,
- numNetworkBuffers);
+ numNetworkBuffers / 2, numNetworkBuffers);
condition2.countDown();
SortMergeResultPartition partition2 =
@@ -547,7 +542,7 @@ class SortMergeResultPartitionTest {
int numBuffers = useHashDataBuffer ? 100 : 15;
int numSubpartitions = 2;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
SortMergeResultPartition partition =
createSortMergedPartition(numSubpartitions, bufferPool);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java
index 1652f80fe91..23280833d0b 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/GateBuffersSpecTest.java
@@ -43,16 +43,17 @@ class GateBuffersSpecTest {
int numInputChannels = 499;
GateBuffersSpec gateBuffersSpec = createGateBuffersSpec(numInputChannels, partitionType);
+ int minFloating = 1;
+ int maxFloating = 8;
int numExclusivePerChannel = 2;
- int expectedBuffersPerGate = 999;
- int maxBuffersPerGate = 1006;
+ int targetTotalBuffersPerGate = 1006;
checkBuffersInGate(
gateBuffersSpec,
+ minFloating,
+ maxFloating,
numExclusivePerChannel,
- expectedBuffersPerGate,
- expectedBuffersPerGate,
- maxBuffersPerGate);
+ targetTotalBuffersPerGate);
}
@ParameterizedTest
@@ -62,16 +63,17 @@ class GateBuffersSpecTest {
GateBuffersSpec gateBuffersSpec = createGateBuffersSpec(numInputChannels, partitionType);
boolean isPipeline = isPipelinedOrHybridResultPartition(partitionType);
+ int minFloating = isPipeline ? 1 : 500;
+ int maxFloating = isPipelinedOrHybridResultPartition(partitionType) ? 8 : 508;
int numExclusivePerChannel = isPipelinedOrHybridResultPartition(partitionType) ? 2 : 1;
- int expectedBuffersPerGate = isPipeline ? 1001 : 1000;
- int maxBuffersPerGate = 1008;
+ int targetTotalBuffersPerGate = 1008;
checkBuffersInGate(
gateBuffersSpec,
+ minFloating,
+ maxFloating,
numExclusivePerChannel,
- expectedBuffersPerGate,
- expectedBuffersPerGate,
- maxBuffersPerGate);
+ targetTotalBuffersPerGate);
}
@ParameterizedTest
@@ -80,17 +82,17 @@ class GateBuffersSpecTest {
int numInputChannels = 999;
GateBuffersSpec gateBuffersSpec = createGateBuffersSpec(numInputChannels, partitionType);
+ int minFloating = 1;
+ int maxFloating = isPipelinedOrHybridResultPartition(partitionType) ? 8 : 1007;
int numExclusivePerChannel = isPipelinedOrHybridResultPartition(partitionType) ? 2 : 1;
- int expectedBuffersPerGate =
- isPipelinedOrHybridResultPartition(partitionType) ? 1999 : 1000;
- int maxBuffersPerGate = 2006;
+ int targetTotalBuffersPerGate = 2006;
checkBuffersInGate(
gateBuffersSpec,
+ minFloating,
+ maxFloating,
numExclusivePerChannel,
- expectedBuffersPerGate,
- expectedBuffersPerGate,
- maxBuffersPerGate);
+ targetTotalBuffersPerGate);
}
@ParameterizedTest
@@ -100,16 +102,17 @@ class GateBuffersSpecTest {
GateBuffersSpec gateBuffersSpec = createGateBuffersSpec(numInputChannels, partitionType);
boolean isPipeline = isPipelinedOrHybridResultPartition(partitionType);
+ int minFloating = isPipeline ? 1 : 1000;
+ int maxFloating = isPipeline ? 8 : numInputChannels * 2 + 8;
int numExclusivePerChannel = isPipeline ? 2 : 0;
- int expectedBuffersPerGate = isPipeline ? 2001 : 1000;
- int maxBuffersPerGate = 2008;
+ int targetTotalBuffersPerGate = 2008;
checkBuffersInGate(
gateBuffersSpec,
+ minFloating,
+ maxFloating,
numExclusivePerChannel,
- expectedBuffersPerGate,
- expectedBuffersPerGate,
- maxBuffersPerGate);
+ targetTotalBuffersPerGate);
}
@ParameterizedTest
@@ -121,16 +124,17 @@ class GateBuffersSpecTest {
createGateBuffersSpec(
numInputChannels, partitionType, numExclusiveBuffersPerChannel);
- int minBuffersPerGate = 1;
- int maxBuffersPerGate = 8;
+ int minFloating = 1;
+ int maxFloating = 8;
int numExclusivePerChannel = 0;
+ int targetTotalBuffersPerGate = 8;
checkBuffersInGate(
gateBuffersSpec,
+ minFloating,
+ maxFloating,
numExclusivePerChannel,
- minBuffersPerGate,
- minBuffersPerGate,
- maxBuffersPerGate);
+ targetTotalBuffersPerGate);
}
@ParameterizedTest
@@ -175,15 +179,16 @@ class GateBuffersSpecTest {
private static void checkBuffersInGate(
GateBuffersSpec gateBuffersSpec,
+ int minFloating,
+ int maxFloating,
int numExclusivePerChannel,
- int expectedBuffersPerGate,
- int minBuffersPerGate,
- int maxBuffersPerGate) {
+ int targetTotalBuffersPerGate) {
+ assertThat(gateBuffersSpec.getRequiredFloatingBuffers()).isEqualTo(minFloating);
+ assertThat(gateBuffersSpec.getTotalFloatingBuffers()).isEqualTo(maxFloating);
assertThat(gateBuffersSpec.getEffectiveExclusiveBuffersPerChannel())
.isEqualTo(numExclusivePerChannel);
- assertThat(gateBuffersSpec.getExpectedBuffersPerGate()).isEqualTo(expectedBuffersPerGate);
- assertThat(gateBuffersSpec.getMinBuffersPerGate()).isEqualTo(minBuffersPerGate);
- assertThat(gateBuffersSpec.getMaxBuffersPerGate()).isEqualTo(maxBuffersPerGate);
+ assertThat(gateBuffersSpec.targetTotalBuffersPerGate())
+ .isEqualTo(targetTotalBuffersPerGate);
}
private static GateBuffersSpec createGateBuffersSpec(
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannelTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannelTest.java
index 3e3efc07b6c..fdd931cae10 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannelTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannelTest.java
@@ -164,7 +164,6 @@ class LocalInputChannelTest {
.setBufferPoolFactory(
() ->
networkBuffers.createBufferPool(
- producerBufferPoolSize,
producerBufferPoolSize,
producerBufferPoolSize,
parallelism,
@@ -204,8 +203,7 @@ class LocalInputChannelTest {
i,
parallelism,
numberOfBuffersPerChannel,
- networkBuffers.createBufferPool(
- parallelism, parallelism, parallelism),
+ networkBuffers.createBufferPool(parallelism, parallelism),
partitionManager,
new TaskEventDispatcher(),
partitionIds);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
index 8d5448c8679..ba73327a73c 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannelTest.java
@@ -127,7 +127,7 @@ class RemoteInputChannelTest {
try {
SingleInputGate inputGate =
new SingleInputGateBuilder()
- .setBufferPoolFactory(networkBufferPool.createBufferPool(1, 1, 1))
+ .setBufferPoolFactory(networkBufferPool.createBufferPool(1, 1))
.build();
inputGate.setup();
RemoteInputChannel channel =
@@ -487,9 +487,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- spy(
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
+ spy(networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -656,9 +654,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- spy(
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
+ spy(networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -745,9 +741,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- spy(
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
+ spy(networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -856,9 +850,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- spy(
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers));
+ spy(networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers));
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputGate.requestPartitions();
@@ -932,8 +924,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
+ networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
buffer = checkNotNull(bufferPool.requestBuffer());
@@ -988,8 +979,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
+ networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1060,8 +1050,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
+ networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1123,8 +1112,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
+ networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1190,8 +1178,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
final BufferPool bufferPool =
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
+ networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1273,8 +1260,7 @@ class RemoteInputChannelTest {
Throwable thrown = null;
try {
BufferPool bufferPool =
- networkBufferPool.createBufferPool(
- numFloatingBuffers, numFloatingBuffers, numFloatingBuffers);
+ networkBufferPool.createBufferPool(numFloatingBuffers, numFloatingBuffers);
inputGate.setBufferPool(bufferPool);
inputGate.setupChannels();
inputChannel.requestSubpartitions();
@@ -1725,7 +1711,7 @@ class RemoteInputChannelTest {
SingleInputGate inputGate =
new SingleInputGateBuilder()
.setChannelFactory(InputChannelBuilder::buildRemoteChannel)
- .setBufferPoolFactory(networkBufferPool.createBufferPool(1, 1, 4))
+ .setBufferPoolFactory(networkBufferPool.createBufferPool(1, 4))
.setSegmentProvider(networkBufferPool)
.build();
inputGate.setup();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
index 47111f8ae81..c7f14e110e4 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateBuilder.java
@@ -117,7 +117,6 @@ public class SingleInputGateBuilder {
SingleInputGateFactory.createBufferPoolFactory(
environment.getNetworkBufferPool(),
1,
- 1,
config.floatingNetworkBuffersPerGate());
this.segmentProvider = environment.getNetworkBufferPool();
return this;
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java
index bb0b4ea5c38..9d549763a0b 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGateTest.java
@@ -173,7 +173,7 @@ class SingleInputGateTest extends InputGateTestBase {
// after setup
assertThat(inputGate.getBufferPool()).isNotNull();
- assertThat(inputGate.getBufferPool().getExpectedNumberOfMemorySegments()).isEqualTo(1);
+ assertThat(inputGate.getBufferPool().getNumberOfRequiredMemorySegments()).isEqualTo(1);
for (InputChannel inputChannel : inputGate.inputChannels()) {
if (inputChannel instanceof RemoteRecoveredInputChannel) {
assertThat(
@@ -190,7 +190,7 @@ class SingleInputGateTest extends InputGateTestBase {
inputGate.convertRecoveredInputChannels();
assertThat(inputGate.getBufferPool()).isNotNull();
- assertThat(inputGate.getBufferPool().getExpectedNumberOfMemorySegments()).isEqualTo(1);
+ assertThat(inputGate.getBufferPool().getNumberOfRequiredMemorySegments()).isEqualTo(1);
for (InputChannel inputChannel : inputGate.inputChannels()) {
if (inputChannel instanceof RemoteInputChannel) {
assertThat(((RemoteInputChannel) inputChannel).getNumberOfAvailableBuffers())
@@ -971,8 +971,7 @@ class SingleInputGateTest extends InputGateTestBase {
// test setup
gate.setup();
assertThat(gate.getBufferPool()).isNotNull();
- assertThat(gate.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(gate.getInputChannels().size() * 2 + 1);
+ assertThat(gate.getBufferPool().getNumberOfRequiredMemorySegments()).isEqualTo(1);
gate.finishReadRecoveredState();
while (!gate.getStateConsumedFuture().isDone()) {
@@ -1228,8 +1227,8 @@ class SingleInputGateTest extends InputGateTestBase {
void testCalculateInputGateNetworkBuffers() throws Exception {
verifyBuffersInBufferPool(true, 2);
verifyBuffersInBufferPool(false, 2);
- verifyBuffersInBufferPool(true, 100);
- verifyBuffersInBufferPool(false, 100);
+ verifyBuffersInBufferPool(true, 500);
+ verifyBuffersInBufferPool(false, 500);
}
// ---------------------------------------------------------------------------------------------
@@ -1273,17 +1272,20 @@ class SingleInputGateTest extends InputGateTestBase {
}
}
- int maxBuffersPerGate = 2 * partitionIds.length * subpartitionRandSize + 8;
- int minBuffersPerGate;
- if (maxBuffersPerGate >= expectMaxRequiredBuffersPerGate.get()) {
- minBuffersPerGate = expectMaxRequiredBuffersPerGate.get();
+ int targetTotalBuffersPerGate = 2 * partitionIds.length * subpartitionRandSize + 8;
+ int requiredFloatingBuffersPerGate;
+ int totalFloatingBuffersPerGate;
+ if (targetTotalBuffersPerGate >= expectMaxRequiredBuffersPerGate.get()) {
+ requiredFloatingBuffersPerGate = expectMaxRequiredBuffersPerGate.get();
+ totalFloatingBuffersPerGate = targetTotalBuffersPerGate;
} else {
- minBuffersPerGate = 2 * partitionIds.length * subpartitionRandSize + 1;
+ requiredFloatingBuffersPerGate = 1;
+ totalFloatingBuffersPerGate = 8;
}
- assertThat(gate.getBufferPool().getExpectedNumberOfMemorySegments())
- .isEqualTo(minBuffersPerGate);
+ assertThat(gate.getBufferPool().getNumberOfRequiredMemorySegments())
+ .isEqualTo(requiredFloatingBuffersPerGate);
assertThat(gate.getBufferPool().getMaxNumberOfMemorySegments())
- .isEqualTo(maxBuffersPerGate);
+ .isEqualTo(totalFloatingBuffersPerGate);
}
private static List<InputChannel> getInputChannelsInPartition(
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsMemoryDataManagerTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsMemoryDataManagerTest.java
index 55003cd5734..5be52e7d471 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsMemoryDataManagerTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsMemoryDataManagerTest.java
@@ -222,8 +222,7 @@ class HsMemoryDataManagerTest {
CompletableFuture<Void> triggerGlobalDecision = new CompletableFuture<>();
NetworkBufferPool networkBufferPool = new NetworkBufferPool(maxBuffers, bufferSize);
- BufferPool bufferPool =
- networkBufferPool.createBufferPool(requiredBuffers, requiredBuffers, maxBuffers);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(requiredBuffers, maxBuffers);
assertThat(bufferPool.getNumBuffers()).isEqualTo(maxBuffers);
HsSpillingStrategy spillingStrategy =
@@ -238,8 +237,7 @@ class HsMemoryDataManagerTest {
.build();
createMemoryDataManager(spillingStrategy, bufferPool);
- networkBufferPool.createBufferPool(
- maxBuffers - requiredBuffers, maxBuffers - requiredBuffers, maxBuffers);
+ networkBufferPool.createBufferPool(maxBuffers - requiredBuffers, maxBuffers);
assertThat(bufferPool.getNumBuffers()).isEqualTo(requiredBuffers);
assertThatFuture(triggerGlobalDecision).eventuallySucceeds();
@@ -255,7 +253,7 @@ class HsMemoryDataManagerTest {
private HsMemoryDataManager createMemoryDataManager(
HsSpillingStrategy spillStrategy, HsFileDataIndex fileDataIndex) throws Exception {
NetworkBufferPool networkBufferPool = new NetworkBufferPool(NUM_BUFFERS, bufferSize);
- BufferPool bufferPool = networkBufferPool.createBufferPool(poolSize, poolSize, poolSize);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(poolSize, poolSize);
return createMemoryDataManager(bufferPool, spillStrategy, fileDataIndex);
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsResultPartitionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsResultPartitionTest.java
index 13403836dc1..faaf6c3c30a 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsResultPartitionTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsResultPartitionTest.java
@@ -124,7 +124,7 @@ class HsResultPartitionTest {
int numRecords = 1000;
Random random = new Random();
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
try (HsResultPartition partition = createHsResultPartition(numSubpartitions, bufferPool)) {
Queue<Tuple2<ByteBuffer, Buffer.DataType>>[] dataWritten = new Queue[numSubpartitions];
@@ -200,7 +200,7 @@ class HsResultPartitionTest {
@Test
void testBroadcastEvent() throws Exception {
final int numBuffers = 1;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
try (HsResultPartition resultPartition = createHsResultPartition(2, bufferPool)) {
resultPartition.broadcastEvent(EndOfPartitionEvent.INSTANCE, false);
// broadcast event does not request buffer
@@ -238,7 +238,7 @@ class HsResultPartitionTest {
final int targetSubpartition = 0;
final Random random = new Random();
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
try (HsResultPartition resultPartition = createHsResultPartition(2, bufferPool)) {
List<ByteBuffer> dataWritten = new ArrayList<>();
for (int i = 0; i < numRecords; i++) {
@@ -293,7 +293,7 @@ class HsResultPartitionTest {
final int numConsumers = 2;
final Random random = new Random();
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
try (HsResultPartition resultPartition = createHsResultPartition(2, bufferPool, true)) {
List<ByteBuffer> dataWritten = new ArrayList<>();
for (int i = 0; i < numRecords; i++) {
@@ -344,7 +344,7 @@ class HsResultPartitionTest {
void testClose() throws Exception {
final int numBuffers = 1;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
HsResultPartition partition = createHsResultPartition(1, bufferPool);
partition.close();
@@ -357,7 +357,7 @@ class HsResultPartitionTest {
final int numSubpartitions = 2;
final int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
HsResultPartition partition =
createHsResultPartition(
numSubpartitions,
@@ -386,7 +386,7 @@ class HsResultPartitionTest {
@Test
void testCreateSubpartitionViewAfterRelease() throws Exception {
final int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
HsResultPartition resultPartition = createHsResultPartition(2, bufferPool);
resultPartition.release();
assertThatThrownBy(
@@ -400,7 +400,7 @@ class HsResultPartitionTest {
@Test
void testCreateSubpartitionViewLostData() throws Exception {
final int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
HsResultPartition resultPartition = createHsResultPartition(2, bufferPool);
IOUtils.deleteFilesRecursively(tempDataPath);
assertThatThrownBy(
@@ -416,7 +416,7 @@ class HsResultPartitionTest {
final int numBuffers = 2;
final int numSubpartitions = 1;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
HsResultPartition partition =
createHsResultPartition(
numSubpartitions,
@@ -439,7 +439,7 @@ class HsResultPartitionTest {
@Test
void testMetricsUpdate() throws Exception {
- BufferPool bufferPool = globalPool.createBufferPool(3, 3, 3);
+ BufferPool bufferPool = globalPool.createBufferPool(3, 3);
try (HsResultPartition partition = createHsResultPartition(2, bufferPool)) {
partition.emitRecord(ByteBuffer.allocate(bufferSize), 0);
partition.broadcastRecord(ByteBuffer.allocate(bufferSize));
@@ -458,7 +458,7 @@ class HsResultPartitionTest {
@Test
void testSelectiveSpillingStrategyRegisterMultipleConsumer() throws Exception {
final int numSubpartitions = 2;
- BufferPool bufferPool = globalPool.createBufferPool(2, 2, 2);
+ BufferPool bufferPool = globalPool.createBufferPool(2, 2);
try (HsResultPartition partition =
createHsResultPartition(
2,
@@ -483,7 +483,7 @@ class HsResultPartitionTest {
@Test
void testFullSpillingStrategyRegisterMultipleConsumer() throws Exception {
final int numSubpartitions = 2;
- BufferPool bufferPool = globalPool.createBufferPool(2, 2, 2);
+ BufferPool bufferPool = globalPool.createBufferPool(2, 2);
try (HsResultPartition partition =
createHsResultPartition(
2,
@@ -506,7 +506,7 @@ class HsResultPartitionTest {
@Test
void testMetricsUpdateForBroadcastOnlyResultPartition() throws Exception {
- BufferPool bufferPool = globalPool.createBufferPool(3, 3, 3);
+ BufferPool bufferPool = globalPool.createBufferPool(3, 3);
try (HsResultPartition partition = createHsResultPartition(2, bufferPool, true)) {
partition.broadcastRecord(ByteBuffer.allocate(bufferSize));
assertThat(taskIOMetricGroup.getNumBuffersOutCounter().getCount()).isEqualTo(1);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionViewTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionViewTest.java
index 15679e0ad1f..99b9d3ba700 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionViewTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/HsSubpartitionViewTest.java
@@ -77,7 +77,7 @@ class HsSubpartitionViewTest {
void testDeadLock(@TempDir Path dataFilePath) throws Exception {
final int bufferSize = 16;
NetworkBufferPool networkBufferPool = new NetworkBufferPool(10, bufferSize);
- BufferPool bufferPool = networkBufferPool.createBufferPool(10, 10, 10);
+ BufferPool bufferPool = networkBufferPool.createBufferPool(10, 10);
HsSubpartitionConsumer subpartitionView = createSubpartitionView();
CompletableFuture<Void> acquireWriteLock = new CompletableFuture<>();
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java
index 865464fc1e0..ec102c1489a 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartitionTest.java
@@ -18,7 +18,6 @@
package org.apache.flink.runtime.io.network.partition.hybrid.tiered.shuffle;
-import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.NettyShuffleEnvironmentOptions.CompressionCodec;
import org.apache.flink.runtime.executiongraph.IOMetrics;
@@ -49,6 +48,7 @@ import org.apache.flink.runtime.util.NoOpTierShuffleDescriptor;
import org.apache.flink.util.concurrent.ExecutorThreadFactory;
import org.apache.flink.util.concurrent.IgnoreShutdownRejectedExecutionHandler;
+import org.apache.commons.lang3.tuple.Pair;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -116,7 +116,7 @@ class TieredResultPartitionTest {
void testClose() throws Exception {
final int numBuffers = 1;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
TieredResultPartition partition = createTieredStoreResultPartition(1, bufferPool, false);
partition.close();
@@ -128,7 +128,7 @@ class TieredResultPartitionTest {
final int numSubpartitions = 2;
final int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
TieredResultPartition partition =
createTieredStoreResultPartition(numSubpartitions, bufferPool, false);
@@ -149,7 +149,7 @@ class TieredResultPartitionTest {
void testMinMaxNetworkBuffersTieredResultPartition() {
int numSubpartitions = 105;
int tieredStorageTotalExclusiveBufferNum = 103;
- Tuple3<Integer, Integer, Integer> minMaxNetworkBuffers =
+ Pair<Integer, Integer> minMaxNetworkBuffers =
getMinMaxNetworkBuffersPerResultPartition(
100,
5,
@@ -159,15 +159,14 @@ class TieredResultPartitionTest {
true,
tieredStorageTotalExclusiveBufferNum,
ResultPartitionType.HYBRID_SELECTIVE);
- assertThat(minMaxNetworkBuffers.f0).isEqualTo(tieredStorageTotalExclusiveBufferNum);
- assertThat(minMaxNetworkBuffers.f1).isEqualTo(8);
- assertThat(minMaxNetworkBuffers.f2).isEqualTo(Integer.MAX_VALUE);
+ assertThat(minMaxNetworkBuffers.getLeft()).isEqualTo(tieredStorageTotalExclusiveBufferNum);
+ assertThat(minMaxNetworkBuffers.getRight()).isEqualTo(Integer.MAX_VALUE);
}
@Test
void testCreateSubpartitionViewAfterRelease() throws Exception {
final int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
TieredResultPartition resultPartition =
createTieredStoreResultPartition(2, bufferPool, false);
resultPartition.release();
@@ -181,7 +180,7 @@ class TieredResultPartitionTest {
@Test
void testEmitRecords() throws Exception {
- BufferPool bufferPool = globalPool.createBufferPool(3, 3, 3);
+ BufferPool bufferPool = globalPool.createBufferPool(3, 3);
int bufferSize = NETWORK_BUFFER_SIZE;
try (TieredResultPartition partition =
createTieredStoreResultPartition(2, bufferPool, false)) {
@@ -193,7 +192,7 @@ class TieredResultPartitionTest {
@Test
void testMetricsUpdateForBroadcastOnlyResultPartition() throws Exception {
- BufferPool bufferPool = globalPool.createBufferPool(3, 3, 3);
+ BufferPool bufferPool = globalPool.createBufferPool(3, 3);
int bufferSize = NETWORK_BUFFER_SIZE;
try (TieredResultPartition partition =
createTieredStoreResultPartition(2, bufferPool, true)) {
@@ -207,7 +206,7 @@ class TieredResultPartitionTest {
final int numBuffers = 20;
final int numRecords = numBuffers / 2;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, 1, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(1, numBuffers);
TieredResultPartition resultPartition =
createTieredStoreResultPartitionWithStorageManager(1, bufferPool, false);
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/HashBufferAccumulatorTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/HashBufferAccumulatorTest.java
index 3ee91645a0c..7e32acfb155 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/HashBufferAccumulatorTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/HashBufferAccumulatorTest.java
@@ -184,8 +184,7 @@ class HashBufferAccumulatorTest {
private TieredStorageMemoryManagerImpl createStorageMemoryManager(int numBuffersInBufferPool)
throws IOException {
BufferPool bufferPool =
- globalPool.createBufferPool(
- numBuffersInBufferPool, numBuffersInBufferPool, numBuffersInBufferPool);
+ globalPool.createBufferPool(numBuffersInBufferPool, numBuffersInBufferPool);
TieredStorageMemoryManagerImpl storageMemoryManager =
new TieredStorageMemoryManagerImpl(NUM_BUFFERS_TRIGGER_FLUSH_RATIO, true);
storageMemoryManager.setup(
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java
index 798ee7d7bc4..ed093c2b9d7 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/SortBufferAccumulatorTest.java
@@ -249,8 +249,7 @@ class SortBufferAccumulatorTest {
private TieredStorageMemoryManagerImpl createStorageMemoryManager(int numBuffersInBufferPool)
throws IOException {
BufferPool bufferPool =
- globalPool.createBufferPool(
- numBuffersInBufferPool, numBuffersInBufferPool, numBuffersInBufferPool);
+ globalPool.createBufferPool(numBuffersInBufferPool, numBuffersInBufferPool);
TieredStorageMemoryManagerImpl storageMemoryManager =
new TieredStorageMemoryManagerImpl(NUM_BUFFERS_TRIGGER_FLUSH_RATIO, true);
storageMemoryManager.setup(
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImplTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImplTest.java
index 345d9d59f3f..395713e48ab 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImplTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageMemoryManagerImplTest.java
@@ -76,7 +76,7 @@ class TieredStorageMemoryManagerImplTest {
void testRequestAndRecycleBuffers() throws IOException {
int numBuffers = 1;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
TieredStorageMemoryManagerImpl storageMemoryManager =
createStorageMemoryManager(
bufferPool,
@@ -94,7 +94,7 @@ class TieredStorageMemoryManagerImplTest {
void testRecycleBuffersAfterPoolSizeDecreased() throws IOException {
int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, 1, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(1, numBuffers);
TieredStorageMemoryManagerImpl storageMemoryManager =
createStorageMemoryManager(
bufferPool,
@@ -117,7 +117,7 @@ class TieredStorageMemoryManagerImplTest {
void testRecycleBuffersAfterReleased() throws IOException {
int numBuffers = 10;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
TieredStorageMemoryManagerImpl storageMemoryManager =
createStorageMemoryManager(
bufferPool,
@@ -294,7 +294,7 @@ class TieredStorageMemoryManagerImplTest {
final int numBuffers = 5;
final int guaranteedReclaimableBuffers = 3;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
TieredStorageMemoryManagerImpl storageMemoryManager =
createStorageMemoryManager(
bufferPool,
@@ -320,7 +320,7 @@ class TieredStorageMemoryManagerImplTest {
@Test
void testRelease() throws IOException {
int numBuffers = 5;
- BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers, numBuffers);
+ BufferPool bufferPool = globalPool.createBufferPool(numBuffers, numBuffers);
TieredStorageMemoryManagerImpl storageMemoryManager =
createStorageMemoryManager(
@@ -354,8 +354,7 @@ class TieredStorageMemoryManagerImplTest {
int numBuffersInBufferPool, List<TieredStorageMemorySpec> storageMemorySpecs)
throws IOException {
BufferPool bufferPool =
- globalPool.createBufferPool(
- numBuffersInBufferPool, numBuffersInBufferPool, numBuffersInBufferPool);
+ globalPool.createBufferPool(numBuffersInBufferPool, numBuffersInBufferPool);
return createStorageMemoryManager(bufferPool, storageMemorySpecs);
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java
index b0ab84f86b0..c56352d23cc 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageSortBufferTest.java
@@ -126,8 +126,7 @@ class TieredStorageSortBufferTest {
int numBuffersForSort = 20;
NetworkBufferPool globalPool = new NetworkBufferPool(bufferPoolSize, BUFFER_SIZE_BYTES);
- BufferPool bufferPool =
- globalPool.createBufferPool(bufferPoolSize, bufferPoolSize, bufferPoolSize);
+ BufferPool bufferPool = globalPool.createBufferPool(bufferPoolSize, bufferPoolSize);
LinkedList<MemorySegment> segments = new LinkedList<>();
for (int i = 0; i < numBuffersForSort; ++i) {
@@ -157,8 +156,7 @@ class TieredStorageSortBufferTest {
Random random = new Random(1234);
NetworkBufferPool globalPool = new NetworkBufferPool(bufferPoolSize, bufferSizeBytes);
- BufferPool bufferPool =
- globalPool.createBufferPool(bufferPoolSize, bufferPoolSize, bufferPoolSize);
+ BufferPool bufferPool = globalPool.createBufferPool(bufferPoolSize, bufferPoolSize);
LinkedList<MemorySegment> segments = new LinkedList<>();
for (int i = 0; i < numBuffersForSort; ++i) {
@@ -262,8 +260,7 @@ class TieredStorageSortBufferTest {
private static TieredStorageSortBuffer createDataBuffer(
int bufferPoolSize, int numSubpartitions) throws Exception {
NetworkBufferPool globalPool = new NetworkBufferPool(bufferPoolSize, BUFFER_SIZE_BYTES);
- BufferPool bufferPool =
- globalPool.createBufferPool(bufferPoolSize, bufferPoolSize, bufferPoolSize);
+ BufferPool bufferPool = globalPool.createBufferPool(bufferPoolSize, bufferPoolSize);
LinkedList<MemorySegment> segments = new LinkedList<>();
for (int i = 0; i < bufferPoolSize; ++i) {
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/shuffle/NettyShuffleUtilsTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/shuffle/NettyShuffleUtilsTest.java
index 7c9e22c319b..3048937b608 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/shuffle/NettyShuffleUtilsTest.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/shuffle/NettyShuffleUtilsTest.java
@@ -29,6 +29,8 @@ import org.apache.flink.runtime.io.network.NettyShuffleEnvironment;
import org.apache.flink.runtime.io.network.NettyShuffleEnvironmentBuilder;
import org.apache.flink.runtime.io.network.partition.ResultPartition;
import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.io.network.partition.consumer.InputChannel;
+import org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel;
import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate;
import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateBuilder;
import org.apache.flink.runtime.jobgraph.IntermediateDataSetID;
@@ -212,12 +214,18 @@ class NettyShuffleUtilsTest {
}
inputGate.convertRecoveredInputChannels();
- return inputGate.getBufferPool().getMaxNumberOfMemorySegments();
+ int ret = 0;
+ for (InputChannel ch : inputGate.inputChannels()) {
+ RemoteInputChannel rChannel = (RemoteInputChannel) ch;
+ ret += rChannel.getNumberOfAvailableBuffers();
+ }
+ ret += inputGate.getBufferPool().getMaxNumberOfMemorySegments();
+ return ret;
}
private int calculateBuffersConsumption(ResultPartition partition) {
if (!partition.getPartitionType().canBePipelinedConsumed()) {
- return partition.getBufferPool().getExpectedNumberOfMemorySegments();
+ return partition.getBufferPool().getNumberOfRequiredMemorySegments();
} else {
return partition.getBufferPool().getMaxNumberOfMemorySegments();
}
diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ChannelPersistenceITCase.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ChannelPersistenceITCase.java
index cd1cc1ee66a..30b9aed26af 100644
--- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ChannelPersistenceITCase.java
+++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ChannelPersistenceITCase.java
@@ -189,7 +189,6 @@ class ChannelPersistenceITCase {
.setBufferPoolFactory(
() ->
networkBufferPool.createBufferPool(
- numberOfSubpartitions,
numberOfSubpartitions,
Integer.MAX_VALUE,
numberOfSubpartitions,
@@ -207,7 +206,7 @@ class ChannelPersistenceITCase {
.setChannelFactory(InputChannelBuilder::buildRemoteRecoveredChannel)
.setBufferPoolFactory(
networkBufferPool.createBufferPool(
- numberOfChannels, numberOfChannels, Integer.MAX_VALUE))
+ numberOfChannels, Integer.MAX_VALUE))
.setSegmentProvider(networkBufferPool)
.setNumberOfChannels(numberOfChannels)
.build();
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/AlignedCheckpointsMassiveRandomTest.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/AlignedCheckpointsMassiveRandomTest.java
index 6f15a3332bf..fe7cb56fca0 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/AlignedCheckpointsMassiveRandomTest.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/AlignedCheckpointsMassiveRandomTest.java
@@ -58,8 +58,8 @@ class AlignedCheckpointsMassiveRandomTest {
try {
networkBufferPool1 = new NetworkBufferPool(100, PAGE_SIZE);
networkBufferPool2 = new NetworkBufferPool(100, PAGE_SIZE);
- BufferPool pool1 = networkBufferPool1.createBufferPool(100, 100, 100);
- BufferPool pool2 = networkBufferPool2.createBufferPool(100, 100, 100);
+ BufferPool pool1 = networkBufferPool1.createBufferPool(100, 100);
+ BufferPool pool2 = networkBufferPool2.createBufferPool(100, 100);
RandomGeneratingInputGate myIG =
new RandomGeneratingInputGate(
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/CheckpointedInputGateTest.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/CheckpointedInputGateTest.java
index a7c5d072291..bfff1d8721a 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/CheckpointedInputGateTest.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/io/checkpointing/CheckpointedInputGateTest.java
@@ -211,7 +211,7 @@ class CheckpointedInputGateTest {
new SingleInputGateBuilder()
.setNumberOfChannels(2)
.setBufferPoolFactory(
- bufferPool.createBufferPool(2, 2, Integer.MAX_VALUE))
+ bufferPool.createBufferPool(2, Integer.MAX_VALUE))
.setSegmentProvider(bufferPool)
.setChannelFactory(InputChannelBuilder::buildRemoteChannel)
.build();
@@ -322,7 +322,7 @@ class CheckpointedInputGateTest {
new SingleInputGateBuilder()
.setBufferPoolFactory(
networkBufferPool.createBufferPool(
- numberOfChannels, numberOfChannels, Integer.MAX_VALUE))
+ numberOfChannels, Integer.MAX_VALUE))
.setSegmentProvider(networkBufferPool)
.setChannelFactory(
(builder, gate) ->
@@ -368,7 +368,7 @@ class CheckpointedInputGateTest {
new SingleInputGateBuilder()
.setBufferPoolFactory(
networkBufferPool.createBufferPool(
- numberOfChannels, numberOfChannels, Integer.MAX_VALUE))
+ numberOfChannels, Integer.MAX_VALUE))
.setSegmentProvider(networkBufferPool)
.setChannelFactory(
(builder, gate) ->
diff --git a/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/TestCheckpointedInputGateBuilder.java b/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/TestCheckpointedInputGateBuilder.java
index f45d35db2f2..11ba54c34bd 100644
--- a/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/TestCheckpointedInputGateBuilder.java
+++ b/flink-streaming-java/src/test/java/org/apache/flink/streaming/util/TestCheckpointedInputGateBuilder.java
@@ -143,8 +143,7 @@ public class TestCheckpointedInputGateBuilder {
.setNumberOfChannels(numChannels)
.setSegmentProvider(networkBufferPool)
.setBufferPoolFactory(
- networkBufferPool.createBufferPool(
- numChannels, numChannels, maxUsedBuffers))
+ networkBufferPool.createBufferPool(numChannels, maxUsedBuffers))
.setChannelStateWriter(channelStateWriter)
.build();
gate.setup();
diff --git a/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java b/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
index e2d66fcbbb4..afd02301bb6 100644
--- a/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
+++ b/flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
@@ -64,10 +64,8 @@ public class SuccessAfterNetworkBuffersFailureITCase extends TestLogger {
private static Configuration getConfiguration() {
Configuration config = new Configuration();
config.set(TaskManagerOptions.MANAGED_MEMORY_SIZE, MemorySize.parse("20m"));
- // Temporarily changes the memory to 4MB since the input gates will request more buffers
- // than before, we will roll back it after the input gate side changes merged.
- config.set(TaskManagerOptions.NETWORK_MEMORY_MIN, MemorySize.ofMebiBytes(4L));
- config.set(TaskManagerOptions.NETWORK_MEMORY_MAX, MemorySize.ofMebiBytes(4L));
+ config.set(TaskManagerOptions.NETWORK_MEMORY_MIN, MemorySize.ofMebiBytes(3L));
+ config.set(TaskManagerOptions.NETWORK_MEMORY_MAX, MemorySize.ofMebiBytes(3L));
return config;
}
(flink) 02/06: Revert "[FLINK-33668][runtime] Close the producer client in case of the result partition is not finished properly"
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
guoweijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 99ad724f664155ed8a6298cf7f76bcebe5b17de4
Author: Yuxin Tan <ta...@gmail.com>
AuthorDate: Fri Jun 21 13:45:34 2024 +0800
Revert "[FLINK-33668][runtime] Close the producer client in case of the result partition is not finished properly"
This reverts commit c1ea971129aa1f91c07cbb250ef4e377a7d67dec.
---
.../partition/hybrid/tiered/shuffle/TieredResultPartition.java | 4 ----
1 file changed, 4 deletions(-)
diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartition.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartition.java
index ee6a0ecb4f1..5272f423348 100644
--- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartition.java
+++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/shuffle/TieredResultPartition.java
@@ -187,10 +187,6 @@ public class TieredResultPartition extends ResultPartition {
@Override
public void close() {
- if (!isFinished()) {
- // Close the producer client in case of the result partition is not finished properly.
- tieredStorageProducerClient.close();
- }
storageMemoryManager.release();
super.close();
}