You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/03/14 11:55:25 UTC
[GitHub] [flink] wsry opened a new pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
wsry opened a new pull request #15199:
URL: https://github.com/apache/flink/pull/15199
## What is the purpose of the change
Currently, sort-merge blocking shuffle implementation uses some unmanaged direct memory and the netty thread for shuffle data read which can lead to direct memory OOM and stress the netty thread with file IO. This patch introduces a separated buffer pool and a separated thread pool for data read. In the following patch, the problem will be totally solved by the IO scheduling mechanism leveraging the introduced buffer pool and thread pool.
## Brief change log
- Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle.
## Verifying this change
This change added tests BatchReadBufferPoolTest.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598064923
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
Review comment:
Because a counter object is more convenient for increase and decrease operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 51a2bfa471abde93d6fa0d040de43f210febc3e1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269)
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d4b7c047feaf021ef4fbbc7bc64880be04474de Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923)
* 33f5e95ff1020b26c602c44537f30cc890c07363 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* ce441b270083d16199d6cac9cec90cfd7977c864 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
}, {
"hash" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323",
"triggerID" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* ce441b270083d16199d6cac9cec90cfd7977c864 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308)
* 2fef21b9a742e587574528bf54a197a21f8c4407 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
}, {
"hash" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323",
"triggerID" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* 2fef21b9a742e587574528bf54a197a21f8c4407 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 225c8fe92a5db0965de7934f945bd56a86966ad9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235)
* aa77fdaa960070379e609f1a17f04a81ac5c9050 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262)
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 225c8fe92a5db0965de7934f945bd56a86966ad9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235)
* aa77fdaa960070379e609f1a17f04a81ac5c9050 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598066194
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
Review comment:
Do you mean when the all buffers are recycled and the allocated buffers reaches 0. If that is the case, the timeout will be triggered in the next round. Currently, we only notify when there is enough buffers or the buffer pool is destroyed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d4b7c047feaf021ef4fbbc7bc64880be04474de Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923)
* 33f5e95ff1020b26c602c44537f30cc890c07363 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c364516ba15d5d72e9b51ffc190bd039e49bd15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914)
* 5d4b7c047feaf021ef4fbbc7bc64880be04474de Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa77fdaa960070379e609f1a17f04a81ac5c9050 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262)
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 51a2bfa471abde93d6fa0d040de43f210febc3e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598063877
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
+
+ if (counter == null && System.currentTimeMillis() - startTime >= requestTimeout) {
+ throw new TimeoutException(
+ String.format(
+ "Can't allocate enough buffers in the given timeout, which means"
+ + " there is a fierce contention for read buffers, please"
+ + " increase %s.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key()));
+ }
+ }
+
+ while (allocated.size() < numBuffersPerRequest) {
+ allocated.add(buffers.poll());
+ }
+
+ if (counter == null) {
+ counter = new Counter();
+ numBuffersAllocated.put(owner, counter);
+ }
+ counter.increase(allocated.size());
+ }
+ return allocated;
+ }
+
+ /**
+ * Recycles the target buffer to this buffer pool. This method should never throw any exception.
+ */
+ public void recycle(MemorySegment segment, Object owner) {
Review comment:
Could we proxy this method to `recycle(Collections.singletonList(segment), owner)` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598064437
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
Review comment:
It is only an empirical value (32M) to let each allocator to use more than 8M memory if there are more available buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599258765
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ while (buffers.size() < numBuffersPerRequest) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ if (System.currentTimeMillis() - startTime >= requestTimeout) {
Review comment:
The wait method has a timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] StephanEwen commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599773493
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
Review comment:
If the defaults are fine, then streaming-only users should also never see problems (I wouldn't expect them to change the defaults).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60d00a17878060352834941cd9e6cccefba4ebf6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879)
* 2e64fa7c8407c4f28eee010630e712dcd177b51d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e64fa7c8407c4f28eee010630e712dcd177b51d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888)
* 9c364516ba15d5d72e9b51ffc190bd039e49bd15 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e64fa7c8407c4f28eee010630e712dcd177b51d Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888)
* 9c364516ba15d5d72e9b51ffc190bd039e49bd15 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599253618
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadIOExecutor.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.runtime.util.Hardware;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link Executor} pool used by batch shuffle for shuffle data read (currently only
+ * used by sort-merge blocking shuffle.
+ */
+@Internal
+public class BatchShuffleReadIOExecutor implements Executor {
+
+ /** Minimum valid number of executor threads. */
+ public static final int MIN_NUM_THREADS = 4;
+
+ /** Maximum valid number of executor threads. */
+ public static final int MAX_NUM_THREADS = Math.max(4, 4 * Hardware.getNumberCPUCores());
+
+ /** Lock for protecting thread-unsafe fields. */
+ private final Object lock = new Object();
+
+ /** Number of threads in this executor pool. */
+ private final int numThreads;
+
+ /** Underlying executor service to run tasks. */
+ @GuardedBy("lock")
+ private ExecutorService executors;
+
+ /**
+ * Whether this executor pool has been destroyed or not. If destroyed, no task can be executed
+ * any more.
+ */
+ @GuardedBy("lock")
+ private boolean destroyed;
+
+ public BatchShuffleReadIOExecutor(int numThreads) {
+ checkArgument(
+ numThreads >= MIN_NUM_THREADS && numThreads <= MAX_NUM_THREADS,
+ String.format(
+ "Illegal thread number, must be no smaller than %d and no larger than %d.",
+ MIN_NUM_THREADS, MAX_NUM_THREADS));
+ this.numThreads = numThreads;
+ }
+
+ public int getNumThreads() {
+ return numThreads;
+ }
+
+ /**
+ * Initializes this executor pool which creates all executor threads, the executor pool must be
+ * initialized before used.
+ */
+ public void initialize() {
Review comment:
I think it can be package private and then we do not need a test for it. `testInitializeAfterDestroy` can be covered by `testExecuteTaskAfterDestroy`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e64fa7c8407c4f28eee010630e712dcd177b51d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r600079683
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
Review comment:
I agree that it should be rare for the streaming to config the memory segment size too large. I will move the check to constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d4b7c047feaf021ef4fbbc7bc64880be04474de Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598062016
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadIOExecutor.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.runtime.util.Hardware;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link Executor} pool used by batch shuffle for shuffle data read (currently only
+ * used by sort-merge blocking shuffle.
+ */
+@Internal
+public class BatchShuffleReadIOExecutor implements Executor {
+
+ /** Minimum valid number of executor threads. */
+ public static final int MIN_NUM_THREADS = 4;
Review comment:
A minimum number of 4 threads is good for IO performance because IO usually need some waiting operation which does not consume cpu cycles. If we only have 1 thread, resources may be wasted. And after the change above, we do not need 32M memory any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
xintongsong commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599453245
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
##########
@@ -513,6 +513,28 @@
+ " the configured min/max size, the min/max size will be used. The exact size of Network Memory can be"
+ " explicitly specified by setting the min/max size to the same value.");
+ /**
+ * Size of direct memory used by blocking shuffle for shuffle data read (currently only used by
+ * sort-merge shuffle).
+ */
+ @Documentation.Section(Documentation.Sections.COMMON_MEMORY)
+ public static final ConfigOption<MemorySize> NETWORK_BATCH_SHUFFLE_READ_MEMORY =
+ key("taskmanager.memory.framework.off-heap.batch-shuffle.size")
+ .memoryType()
+ .defaultValue(MemorySize.parse("32m"))
+ .withDescription(
+ String.format(
+ "Size of memory used by blocking shuffle for shuffle data read "
+ + "(currently only used by sort-merge shuffle). Notes: "
+ + "1) The memory is cut from %s so must be smaller than"
+ + " that, which means you may also need to increase %s "
Review comment:
nit:
```suggestion
+ "1) The memory is cut from '%s' so must be smaller than"
+ " that, which means you may also need to increase '%s' "
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
Review comment:
nit:
```suggestion
"Illegal configuration, config value for '%s' must be no smaller than '%s',"
+ " please increase '%s' to at least %d bytes.",
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.forEach(MemorySegment::free);
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout.
+ */
+ public List<MemorySegment> requestBuffers() throws Exception {
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ while (buffers.size() < numBuffersPerRequest) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ if (System.currentTimeMillis() - startTime >= requestTimeout) {
+ throw new TimeoutException(
+ String.format(
+ "Can't allocate enough buffers in the given timeout, which means"
+ + " there is a fierce contention for read buffers, please"
+ + " increase %s.",
Review comment:
nit:
```suggestion
+ " increase '%s'.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598080749
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleServiceFactory.java
##########
@@ -119,13 +121,34 @@ static NettyShuffleEnvironment createNettyShuffleEnvironment(
config.networkBufferSize(),
config.getRequestSegmentsTimeout());
+ // we create a separated buffer pool here for batch shuffle instead of reusing the network
+ // buffer pool directly to avoid potential side effects of memory contention, for example,
+ // dead lock or "insufficient network buffer" error
+ BatchShuffleReadBufferPool batchShuffleReadBufferPool =
+ new BatchShuffleReadBufferPool(
+ config.batchShuffleReadMemoryBytes(), config.networkBufferSize());
+
+ // we create a separated IO executor pool here for batch shuffle instead of reusing the
+ // TaskManager IO executor pool directly to avoid the potential side effects of execution
+ // contention, for example, too long IO or waiting time leading to starvation or timeout
+ int numThreads =
+ batchShuffleReadBufferPool.getNumTotalBuffers()
+ / Math.max(1, batchShuffleReadBufferPool.getNumBuffersPerRequest());
Review comment:
numTotalBuffers can be 0 if the total bytes is less than the buffer size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-804620175
@zhuzhurk Thanks for the review and comments, I have updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599253364
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadIOExecutor.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.runtime.util.Hardware;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link Executor} pool used by batch shuffle for shuffle data read (currently only
+ * used by sort-merge blocking shuffle.
+ */
+@Internal
+public class BatchShuffleReadIOExecutor implements Executor {
+
+ /** Minimum valid number of executor threads. */
+ public static final int MIN_NUM_THREADS = 4;
+
+ /** Maximum valid number of executor threads. */
+ public static final int MAX_NUM_THREADS = Math.max(4, 4 * Hardware.getNumberCPUCores());
+
+ /** Lock for protecting thread-unsafe fields. */
+ private final Object lock = new Object();
+
+ /** Number of threads in this executor pool. */
+ private final int numThreads;
+
+ /** Underlying executor service to run tasks. */
+ @GuardedBy("lock")
+ private ExecutorService executors;
+
+ /**
+ * Whether this executor pool has been destroyed or not. If destroyed, no task can be executed
+ * any more.
+ */
+ @GuardedBy("lock")
+ private boolean destroyed;
+
+ public BatchShuffleReadIOExecutor(int numThreads) {
+ checkArgument(
+ numThreads >= MIN_NUM_THREADS && numThreads <= MAX_NUM_THREADS,
+ String.format(
+ "Illegal thread number, must be no smaller than %d and no larger than %d.",
+ MIN_NUM_THREADS, MAX_NUM_THREADS));
+ this.numThreads = numThreads;
+ }
+
+ public int getNumThreads() {
Review comment:
it's not needed I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599698053
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.forEach(MemorySegment::free);
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout.
+ */
+ public List<MemorySegment> requestBuffers() throws Exception {
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ while (buffers.size() < numBuffersPerRequest) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ if (System.currentTimeMillis() - startTime >= requestTimeout) {
+ throw new TimeoutException(
+ String.format(
+ "Can't allocate enough buffers in the given timeout, which means"
+ + " there is a fierce contention for read buffers, please"
+ + " increase %s.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key()));
+ }
+ }
+
+ while (allocated.size() < numBuffersPerRequest) {
+ allocated.add(buffers.poll());
+ }
+ }
+ return allocated;
+ }
+
+ /**
+ * Recycles the target buffer to this buffer pool. This method should never throw any exception.
+ */
+ public void recycle(MemorySegment segment) {
+ checkArgument(segment != null, "Buffer must be not null.");
+
+ recycle(Collections.singletonList(segment));
+ }
+
+ /**
+ * Recycles a collection of buffers to this buffer pool. This method should never throw any
+ * exception.
+ */
+ public void recycle(Collection<MemorySegment> segments) {
+ checkArgument(segments != null, "Buffer list must be not null.");
+
+ if (segments.isEmpty()) {
+ return;
+ }
+
+ synchronized (buffers) {
+ checkState(initialized, "Recycling a buffer before initialization.");
+
+ if (destroyed) {
+ return;
+ }
+
+ buffers.addAll(segments);
+ if (buffers.size() >= numBuffersPerRequest) {
+ buffers.notifyAll();
+ }
+ }
+ }
+
+ /** Destroys this buffer pool and after which, no buffer can be allocated any more. */
+ public void destroy() {
+ synchronized (buffers) {
+ destroyed = true;
Review comment:
It has been already cleared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r597382774
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
##########
@@ -513,6 +513,30 @@
+ " the configured min/max size, the min/max size will be used. The exact size of Network Memory can be"
+ " explicitly specified by setting the min/max size to the same value.");
+ /**
+ * Memory used by blocking shuffle for shuffle data read (currently only used by sort-merge
+ * shuffle). The minimum valid value can be configured is 32M.
Review comment:
Not sure why we must require it to be at least 32m? Even if for FRAMEWORK_OFF_HEAP_MEMORY it is not required to be larger than 32m.
Or maybe the smallest allowed value should be BatchShuffleReadBufferPool.NUM_BYTES_PER_REQUEST which is 8m?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadIOExecutor.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.runtime.util.Hardware;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link Executor} pool used by batch shuffle for shuffle data read (currently only
+ * used by sort-merge blocking shuffle.
+ */
+@Internal
+public class BatchShuffleReadIOExecutor implements Executor {
+
+ /** Minimum valid number of executor threads. */
+ public static final int MIN_NUM_THREADS = 4;
Review comment:
is it necessary to add a lower-bound limit to the threads?
IIUC, this further results in a 32m requirement of the shuffle read memory size. maybe 32m as the default values of NETWORK_BATCH_SHUFFLE_READ_MEMORY is enough for out of box experience?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
Review comment:
I can see that most these getters only serves for testing purpose or even not used. Let's we drop the unused ones and mark the ones for testing with `@VisibleForTesting`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
Review comment:
I think `IdentityHashMap` is better in this case.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
Review comment:
I guess `TaskManagerOptions.MEMORY_SEGMENT_SIZE.key()` should be the last but one param of this `String.format`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
Review comment:
I'm not quite sure here, is `buffer.clear()` enough or do we need to explicit return the buffers?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
Review comment:
I think the `@Internal` annotation is not needed because `flink-runtime` is not an api module.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleServiceFactory.java
##########
@@ -119,13 +121,34 @@ static NettyShuffleEnvironment createNettyShuffleEnvironment(
config.networkBufferSize(),
config.getRequestSegmentsTimeout());
+ // we create a separated buffer pool here for batch shuffle instead of reusing the network
+ // buffer pool directly to avoid potential side effects of memory contention, for example,
+ // dead lock or "insufficient network buffer" error
+ BatchShuffleReadBufferPool batchShuffleReadBufferPool =
+ new BatchShuffleReadBufferPool(
+ config.batchShuffleReadMemoryBytes(), config.networkBufferSize());
+
+ // we create a separated IO executor pool here for batch shuffle instead of reusing the
+ // TaskManager IO executor pool directly to avoid the potential side effects of execution
+ // contention, for example, too long IO or waiting time leading to starvation or timeout
+ int numThreads =
+ batchShuffleReadBufferPool.getNumTotalBuffers()
+ / Math.max(1, batchShuffleReadBufferPool.getNumBuffersPerRequest());
Review comment:
the `Math.max` is not needed I guess.
and maybe we can have a method `batchShuffleReadBufferPool.getMaxAcceptableConcurrentRequests()` and direct get it as `numThreads`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
Review comment:
In my understanding, this method is needed for testing cases that `requestTimeout` can happen. However, I did not find any test for the `requestTimeout` except for illegal argument check. But if we do not expose this method, there is no chance to have an illegal `requestTimeout` param.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
Review comment:
is it possible to make `numTotalBuffers` a long type so we do not need such a `Math.min`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
Review comment:
this is inconsistent with the logic `this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE)` in the the constructor. Maybe move this check to the constructor and simplify the calculation of `numTotalBuffers`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadIOExecutor.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.runtime.util.Hardware;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link Executor} pool used by batch shuffle for shuffle data read (currently only
+ * used by sort-merge blocking shuffle.
+ */
+@Internal
+public class BatchShuffleReadIOExecutor implements Executor {
Review comment:
I'm a bit wondering whether we need to implement such an executor, instead of directly the executor:
`executors = Executors.newFixedThreadPool(numThreads, threadFactory)`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
Review comment:
seems `buffers` is not notified when `numBuffersAllocated` is updated.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
Review comment:
why we do expect one requester to have at least 4 concurrent requests at the same time?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
+
+ if (counter == null && System.currentTimeMillis() - startTime >= requestTimeout) {
Review comment:
should exception also be thrown if `counter != null && counter.get() + numBuffersPerRequest <= maxBuffersPerRequester`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
Review comment:
IIUC, this should be `Math.min`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
Review comment:
maybe it should be `numTotalBuffers / numThreads`? And given that `numThreads` equals `numTotalBuffers/numBuffersPerRequest`(i.e. `maxAcceptableConcurrentRequests`), the does not need to aware of `BatchShuffleReadIOExecutor`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598063730
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
Review comment:
The last %s should be %d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33f5e95ff1020b26c602c44537f30cc890c07363 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761)
* 60d00a17878060352834941cd9e6cccefba4ebf6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879)
* 2e64fa7c8407c4f28eee010630e712dcd177b51d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 51a2bfa471abde93d6fa0d040de43f210febc3e1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269)
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* ce441b270083d16199d6cac9cec90cfd7977c864 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r597604195
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
Review comment:
Do we need to check whether totalBytes could be divided by bufferSize without remainder ?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadIOExecutor.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.runtime.util.Hardware;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link Executor} pool used by batch shuffle for shuffle data read (currently only
+ * used by sort-merge blocking shuffle.
+ */
+@Internal
+public class BatchShuffleReadIOExecutor implements Executor {
Review comment:
+1 for this one, it seems `BatchShuffleReadIOExecutor` is a direct wrapper of the fixed thread pool ?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
Review comment:
Why we need use Counter instead of Map<Object, Integer> ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598074585
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
+
+ if (counter == null && System.currentTimeMillis() - startTime >= requestTimeout) {
Review comment:
buffer recycle will always notify the waiting thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599254630
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
Review comment:
Could we make it private and `testInitializeAfterDestroy` can be replaced with a `testRequestBuffersAfterDestroy`?
For those case that need the pool to be initialized, we can use `execute` to trigger the initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e64fa7c8407c4f28eee010630e712dcd177b51d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888)
* 9c364516ba15d5d72e9b51ffc190bd039e49bd15 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598066787
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
Review comment:
I'm afraid we might still need the notification since it is also waiting for the condition `(counter != null
&& counter.get() + numBuffersPerRequest > maxBuffersPerRequester)`, if there is no notification for this condition, after there are enough buffers, the thread might still blocked on this condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598063840
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
Review comment:
I am sure ```buffer.clear()```is enough for direct memory. GC will do the work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
}, {
"hash" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* ce441b270083d16199d6cac9cec90cfd7977c864 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308)
* 2fef21b9a742e587574528bf54a197a21f8c4407 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599700030
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
Review comment:
You are right, I will replace it with if statement though the initialize method is not called frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-805025446
@StephanEwen Thanks for the review and comments.
**About Lazy Initialization**
I agree that initializing the ExecutorService eagerly is better if it already handles lazy thread spawning and termination.
**About Buffer Request Batching**
I have already tested the default configuration with tpcds benchmark. Compared with my previous implementation which uses exclusive buffers and exclusive thread per result partition, there is no performance regression. Besides, batch reading is important for IO performance especially on HDD. **Buffer Request Batching** is a simple way to guarantee that the reader can always get enough buffers for sequential read, otherwise, all readers may get a few buffers and read in parallel which can be bad for performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk closed pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk closed pull request #15199:
URL: https://github.com/apache/flink/pull/15199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-805025446
@StephanEwen Thanks for the review and comments.
**About Lazy Initialization**
I agree that initializing the ExecutorService eagerly is better if it already handles lazy thread spawning and termination.
**About Buffer Request Batching**
I have already tested the default configuration with tpcds benchmark. Compared with my previous implementation which uses exclusive buffers and exclusive thread per result partition, there is no performance regression. Besides, batch reading is important for IO performance especially on HDD. **Buffer Request Batching** is a simple way to guarantee that the reader can always get enough buffers for sequential read, otherwise, all readers may get a few buffers and read in parallel which can be bad for performance.
We may can implement a method which accepts a "size" parameter and can allocate any size batching for the requester. However, we do not need such a method now and implementing it can introduce extra complexity. I think we can implement it when we need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r600139401
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
Review comment:
I totally agree we should not enforce that and there is no such enforcement in the latest version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598681288
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NettyShuffleServiceFactory.java
##########
@@ -119,13 +121,33 @@ static NettyShuffleEnvironment createNettyShuffleEnvironment(
config.networkBufferSize(),
config.getRequestSegmentsTimeout());
+ // we create a separated buffer pool here for batch shuffle instead of reusing the network
+ // buffer pool directly to avoid potential side effects of memory contention, for example,
+ // dead lock or "insufficient network buffer" error
+ BatchShuffleReadBufferPool batchShuffleReadBufferPool =
+ new BatchShuffleReadBufferPool(
+ config.batchShuffleReadMemoryBytes(), config.networkBufferSize());
+
+ // we create a separated IO executor pool here for batch shuffle instead of reusing the
+ // TaskManager IO executor pool directly to avoid the potential side effects of execution
+ // contention, for example, too long IO or waiting time leading to starvation or timeout
+ BatchShuffleReadIOExecutor batchShuffleReadIOExecutor =
+ new BatchShuffleReadIOExecutor(
+ Math.max(
Review comment:
maybe let's do this min and max in the ctor of BatchShuffleReadIOExecutor? Then `BatchShuffleReadIOExecutor.MIN_NUM_THREADS` and `BatchShuffleReadIOExecutor.MAX_NUM_THREADS` can be private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-803275446
@zhuzhurk @gaoyunhaii Thanks for the review and comments. I have updated the PR by pushing a fixup commit. You can take a look.
BTW, I removed the max buffers per requester logic in the fixup commit because the blocking wait occupies a thread which should be used by others. I will implement a similar logic at requester side. (A requester will not request new buffers if too many buffers are already allocated, only when total requested buffers are smaller than the threshold, the data read task will be resubmitted again)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c364516ba15d5d72e9b51ffc190bd039e49bd15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598081685
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
Review comment:
The recycle thread will always notify the lock
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 225c8fe92a5db0965de7934f945bd56a86966ad9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235)
* aa77fdaa960070379e609f1a17f04a81ac5c9050 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262)
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 51a2bfa471abde93d6fa0d040de43f210febc3e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33f5e95ff1020b26c602c44537f30cc890c07363 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096)
* 225c8fe92a5db0965de7934f945bd56a86966ad9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761)
* 60d00a17878060352834941cd9e6cccefba4ebf6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879)
* 2e64fa7c8407c4f28eee010630e712dcd177b51d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33f5e95ff1020b26c602c44537f30cc890c07363 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096)
* 225c8fe92a5db0965de7934f945bd56a86966ad9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598063014
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
Review comment:
numTotalBuffers must be int, because the size of java collections is Integer, we can not avoid checking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 2e64fa7c8407c4f28eee010630e712dcd177b51d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598064204
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
Review comment:
As discussed offline, we will remove the upper bound and use 4 * numBuffersPerRequest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9e2228fabe711bfa14d967084f2580cf51bdf4f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620)
* cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] StephanEwen commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-806924848
Looks good to me.
Can you confirm that when you start a Flink application or cluster with this change, it has no impact at this point?
If yes, then +1 to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 51a2bfa471abde93d6fa0d040de43f210febc3e1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269)
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* ce441b270083d16199d6cac9cec90cfd7977c864 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599701700
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
Review comment:
Doing the check in the initialize method is to avoid potential failure for streaming only user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599709174
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
Review comment:
Because buffer is the smallest data unit used by writing and credit-based back pressure, using the same data unit for reading is an easy way. Besides, the memory will not be wasted because the memory is cut from framework and other component can use the memory. As for configuration, there should be no problem if the memory segment size is not set to large (default 32K). I guess setting this size too large should be rare.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598062812
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
Review comment:
You are right, it only for test currently, because the default 5min timeout is a little long for test. BatchShuffleReadBufferPoolTest#testRequestBuffersTimeout and BatchShuffleReadBufferPoolTest#testNoBufferTimeoutIfHoldingBuffers use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598061122
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
##########
@@ -513,6 +513,30 @@
+ " the configured min/max size, the min/max size will be used. The exact size of Network Memory can be"
+ " explicitly specified by setting the min/max size to the same value.");
+ /**
+ * Memory used by blocking shuffle for shuffle data read (currently only used by sort-merge
+ * shuffle). The minimum valid value can be configured is 32M.
Review comment:
The reason for this is that too small read memory is bad for performance and 32M is not a big memory overhead. Removing this restriction is also acceptable for me. I will remove and add some descriptions like: decreasing this value may impact the performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9e2228fabe711bfa14d967084f2580cf51bdf4f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598066726
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
+
+ if (counter == null && System.currentTimeMillis() - startTime >= requestTimeout) {
Review comment:
It should be guaranteed only one thread can serve a requester at the same time, I can add this constraint to the java doc of this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9e2228fabe711bfa14d967084f2580cf51bdf4f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598065049
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
Review comment:
It was checked it initialize method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
}, {
"hash" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323",
"triggerID" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"triggerType" : "PUSH"
}, {
"hash" : "0d6ada9ddd0847e4306de24b6466bf17581a7842",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15500",
"triggerID" : "0d6ada9ddd0847e4306de24b6466bf17581a7842",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* 0d6ada9ddd0847e4306de24b6466bf17581a7842 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15500)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798894606
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit c9e2228fabe711bfa14d967084f2580cf51bdf4f (Sun Mar 14 11:58:04 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20740).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9e2228fabe711bfa14d967084f2580cf51bdf4f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
}, {
"hash" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323",
"triggerID" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"triggerType" : "PUSH"
}, {
"hash" : "0d6ada9ddd0847e4306de24b6466bf17581a7842",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15500",
"triggerID" : "0d6ada9ddd0847e4306de24b6466bf17581a7842",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* 2fef21b9a742e587574528bf54a197a21f8c4407 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323)
* 0d6ada9ddd0847e4306de24b6466bf17581a7842 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15500)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c364516ba15d5d72e9b51ffc190bd039e49bd15 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914)
* 5d4b7c047feaf021ef4fbbc7bc64880be04474de UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598065608
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
Review comment:
I think not needed. The check is at the start of the while loop. If the pool has been destroyed, the buffers will be cleared and the while condition will be true and the check at the start of the while loop will be reached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-807885922
Rebase master and rerun tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r600138798
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
Review comment:
@StephanEwen Sorry for the misunderstanding of this reply, let us just ignore this. I misunderstood the context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 225c8fe92a5db0965de7934f945bd56a86966ad9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235)
* aa77fdaa960070379e609f1a17f04a81ac5c9050 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-805714093
Hi @StephanEwen, is there any other concerns? If not, we will merge the PR soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c9e2228fabe711bfa14d967084f2580cf51bdf4f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620)
* cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598063366
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
Review comment:
I add this check to initialize instead of constructor to keep the behavior not changed. Because if one have set the memory segment size to larger than 32M, then after he/she upgrades to the new version, exception will be thrown in the constructor even sort-merge shuffle is not enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-805065015
@StephanEwen I have updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-807993958
Thanks for the further reviewing! @StephanEwen @gaoyunhaii
Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598064490
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
Review comment:
As discussed offline, we will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598066440
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
+
+ if (counter == null && System.currentTimeMillis() - startTime >= requestTimeout) {
Review comment:
It is not guaranteed the requester can get the upper bound number of buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598064285
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
Review comment:
After waiting, do we still need to check whether it is destroyed here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599399255
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
Review comment:
I think we might still have a call to `MemorySegment.free()` in case there might be future changes on how memorySegmentFactory allocates off-heap memory on OOM and destroy. We might have a method to cleanup the current `buffer` array to be called in the two places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761)
* 60d00a17878060352834941cd9e6cccefba4ebf6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 225c8fe92a5db0965de7934f945bd56a86966ad9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235)
* aa77fdaa960070379e609f1a17f04a81ac5c9050 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262)
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 51a2bfa471abde93d6fa0d040de43f210febc3e1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-798900245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14620",
"triggerID" : "c9e2228fabe711bfa14d967084f2580cf51bdf4f",
"triggerType" : "PUSH"
}, {
"hash" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14761",
"triggerID" : "cc48211bdf1785f59b98c7a3fc598ce70e4fd7e3",
"triggerType" : "PUSH"
}, {
"hash" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14879",
"triggerID" : "60d00a17878060352834941cd9e6cccefba4ebf6",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"triggerType" : "PUSH"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14911",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "2e64fa7c8407c4f28eee010630e712dcd177b51d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14888",
"triggerID" : "801203146",
"triggerType" : "MANUAL"
}, {
"hash" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14914",
"triggerID" : "9c364516ba15d5d72e9b51ffc190bd039e49bd15",
"triggerType" : "PUSH"
}, {
"hash" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=14923",
"triggerID" : "5d4b7c047feaf021ef4fbbc7bc64880be04474de",
"triggerType" : "PUSH"
}, {
"hash" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15096",
"triggerID" : "33f5e95ff1020b26c602c44537f30cc890c07363",
"triggerType" : "PUSH"
}, {
"hash" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15235",
"triggerID" : "225c8fe92a5db0965de7934f945bd56a86966ad9",
"triggerType" : "PUSH"
}, {
"hash" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15262",
"triggerID" : "aa77fdaa960070379e609f1a17f04a81ac5c9050",
"triggerType" : "PUSH"
}, {
"hash" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98c5f207e1d006b91be50df6ce1b3e8f45fc795b",
"triggerType" : "PUSH"
}, {
"hash" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15269",
"triggerID" : "51a2bfa471abde93d6fa0d040de43f210febc3e1",
"triggerType" : "PUSH"
}, {
"hash" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33",
"triggerType" : "PUSH"
}, {
"hash" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15308",
"triggerID" : "ce441b270083d16199d6cac9cec90cfd7977c864",
"triggerType" : "PUSH"
}, {
"hash" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323",
"triggerID" : "2fef21b9a742e587574528bf54a197a21f8c4407",
"triggerType" : "PUSH"
}, {
"hash" : "0d6ada9ddd0847e4306de24b6466bf17581a7842",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d6ada9ddd0847e4306de24b6466bf17581a7842",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98c5f207e1d006b91be50df6ce1b3e8f45fc795b UNKNOWN
* 3b6c434b676ee0d1aaad00e8ba1fa03c1ddc7f33 UNKNOWN
* 2fef21b9a742e587574528bf54a197a21f8c4407 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=15323)
* 0d6ada9ddd0847e4306de24b6466bf17581a7842 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-807883369
Yes, I can confirm that it has no impact at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598062094
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadIOExecutor.java
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.runtime.util.ExecutorThreadFactory;
+import org.apache.flink.runtime.util.Hardware;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link Executor} pool used by batch shuffle for shuffle data read (currently only
+ * used by sort-merge blocking shuffle.
+ */
+@Internal
+public class BatchShuffleReadIOExecutor implements Executor {
Review comment:
BatchShuffleReadIOExecutor will be Initialized lazily when first used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] StephanEwen commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
StephanEwen commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599499302
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
Review comment:
Never do work in constructing strings that you pass to check methods. That is very expensive (parsing, concatenation, copying) work you do every time, even when the error message isn't needed. Always do this lazily.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
Review comment:
I would suggest to use `Duration` for all timeouts. That eliminates any confusion about time units (is it millis, nanos, seconds?)
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
Review comment:
I wouldn't do any checks here in `initialize()`. That results in late failures much later during execution, rather than at TM startup. Config related problems should all show as early as possible, ideally during TM startup.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.forEach(MemorySegment::free);
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout.
+ */
+ public List<MemorySegment> requestBuffers() throws Exception {
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
Review comment:
`System.currentTimeMillis()` is unstable, the clock can jump. For interval measuring, always use `System.nanoTime()`, which is guaranteed to be stable.
You can also use the `Deadline` class to capture this in an easier way: `Deadline.fromNow(...)`, `Deadline.timeLeft(...)`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
Review comment:
I wouldn't enforce that. Worst case is a few bytes are wasted. Enforcing this means configuration is harder, users need to provide specific values and understand the logic with buffer sizes.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ private static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ private void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.forEach(MemorySegment::free);
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout.
+ */
+ public List<MemorySegment> requestBuffers() throws Exception {
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ while (buffers.size() < numBuffersPerRequest) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ if (System.currentTimeMillis() - startTime >= requestTimeout) {
+ throw new TimeoutException(
+ String.format(
+ "Can't allocate enough buffers in the given timeout, which means"
+ + " there is a fierce contention for read buffers, please"
+ + " increase %s.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key()));
+ }
+ }
+
+ while (allocated.size() < numBuffersPerRequest) {
+ allocated.add(buffers.poll());
+ }
+ }
+ return allocated;
+ }
+
+ /**
+ * Recycles the target buffer to this buffer pool. This method should never throw any exception.
+ */
+ public void recycle(MemorySegment segment) {
+ checkArgument(segment != null, "Buffer must be not null.");
+
+ recycle(Collections.singletonList(segment));
+ }
+
+ /**
+ * Recycles a collection of buffers to this buffer pool. This method should never throw any
+ * exception.
+ */
+ public void recycle(Collection<MemorySegment> segments) {
+ checkArgument(segments != null, "Buffer list must be not null.");
+
+ if (segments.isEmpty()) {
+ return;
+ }
+
+ synchronized (buffers) {
+ checkState(initialized, "Recycling a buffer before initialization.");
+
+ if (destroyed) {
+ return;
+ }
+
+ buffers.addAll(segments);
+ if (buffers.size() >= numBuffersPerRequest) {
+ buffers.notifyAll();
+ }
+ }
+ }
+
+ /** Destroys this buffer pool and after which, no buffer can be allocated any more. */
+ public void destroy() {
+ synchronized (buffers) {
+ destroyed = true;
Review comment:
Maybe clear the buffers list here, to allow GC to reclaim memory earlier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-801203146
@flinkbot run azure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry edited a comment on pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
wsry edited a comment on pull request #15199:
URL: https://github.com/apache/flink/pull/15199#issuecomment-805065015
@StephanEwen I have updated the PR. Is there any other concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r598065309
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ counter = numBuffersAllocated.get(owner);
+
+ if (counter == null && System.currentTimeMillis() - startTime >= requestTimeout) {
Review comment:
I have another concern here in that based on the condition for waiting, it seems a requester (owner) might have multiple threads work for it concurrently, right ? if so, suppose the current thread keep waiting due to another thread work for it holds too much buffer than `maxBuffersPerRequester` for a long time, and at last round, the other thread recycled all the buffers after timeout, then TimeoutException is thrown here. Would this also be expected ? It seems each requester has hold some buffers in this case and no requester are starved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #15199: [FLINK-20740][network] Introduce a separated buffer pool and a separated thread pool for sort-merge blocking shuffle
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #15199:
URL: https://github.com/apache/flink/pull/15199#discussion_r599248454
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/ResultPartitionBuilder.java
##########
@@ -50,6 +52,12 @@
private NetworkBufferPool networkBufferPool = new NetworkBufferPool(2, 1);
+ private BatchShuffleReadBufferPool batchShuffleReadBufferPool =
+ new BatchShuffleReadBufferPool(1024, 32 * 1024);
Review comment:
why set a `totalBytes` to be smaller than `bufferSize` in this default batchShuffleReadBufferPool? Seems it will lead to exceptions if used.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ @VisibleForTesting
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ @VisibleForTesting
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ public int getMaxConcurrentRequests() {
+ return numBuffersPerRequest > 0 ? numTotalBuffers / numBuffersPerRequest : 0;
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %d bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ while (buffers.size() < numBuffersPerRequest) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
+ if (System.currentTimeMillis() - startTime >= requestTimeout) {
Review comment:
seems the timeout will never trigger if available `buffers` does not increase to be larger than `numBuffersPerRequest`, because `buffers` is only notified when it can fulfill a request.
I feel that we should not let the timeout rely on buffer returning events, because in the worst case that no buffer can be returned, timeout will never be triggered.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
Review comment:
Let's annotate it as `@VisibleForTesting` and make it package private.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ }
+
+ @VisibleForTesting
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ @VisibleForTesting
Review comment:
Seems all these `VisibleForTesting` getters can be package private
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/NettyShuffleEnvironmentBuilder.java
##########
@@ -180,7 +187,8 @@ public NettyShuffleEnvironment build() {
compressionCodec,
maxBuffersPerChannel,
sortShuffleMinBuffers,
- sortShuffleMinParallelism),
+ sortShuffleMinParallelism,
+ batchShuffleReadBuffers),
Review comment:
I guess it should be `batchShuffleReadBytes` and it should be placed in ahead of the param `sortShuffleMinBuffers`.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPoolTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Copyright 2012 The Netty Project
+ * Copy from netty 4.1.32.Final
+ *
+ * The Netty Project licenses this file to you under the Apache License,
+ * version 2.0 (the "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at:
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.core.memory.MemorySegment;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BatchShuffleReadBufferPool}. */
+public class BatchShuffleReadBufferPoolTest {
+
+ @Rule public Timeout timeout = new Timeout(60, TimeUnit.SECONDS);
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalTotalBytes() {
+ createBufferPool(0, 1024);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalBufferSize() {
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, 0);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalRequestTimeout() {
+ createBufferPool(0);
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testInitializeAfterDestroy() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ bufferPool.destroy();
+ bufferPool.initialize();
+ }
+
+ @Test
+ public void testLargeTotalBytes() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(Long.MAX_VALUE, 1024);
+ assertEquals(Integer.MAX_VALUE, bufferPool.getNumTotalBuffers());
+ bufferPool.destroy();
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testTotalBytesSmallerThanBufferSize() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(4096, 32 * 1024);
+ try {
+ bufferPool.initialize();
+ } finally {
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testBufferCalculation() {
+ long totalBytes = BatchShuffleReadBufferPool.MIN_TOTAL_BYTES;
+ for (int bufferSize = 4 * 1024; bufferSize <= totalBytes; bufferSize += 1024) {
+ BatchShuffleReadBufferPool bufferPool =
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, bufferSize);
+
+ assertEquals(totalBytes, bufferPool.getTotalBytes());
+ assertEquals(totalBytes / bufferSize, bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() <= bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() > 0);
+ }
+ }
+
+ @Test
+ public void testRequestBuffers() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ List<MemorySegment> buffers = new ArrayList<>();
+
+ try {
+ buffers.addAll(bufferPool.requestBuffers(this));
+ assertEquals(bufferPool.getNumBuffersPerRequest(), buffers.size());
+ } finally {
+ bufferPool.recycle(buffers, this);
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ bufferPool.destroy();
+ }
+ }
+
+ @Test(expected = TimeoutException.class)
+ public void testRequestBuffersTimeout() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(1000);
+ List<MemorySegment> buffers = new ArrayList<>();
+
+ try {
+ for (int i = 0; i < 4; ++i) {
+ buffers.addAll(bufferPool.requestBuffers(this));
+ }
+ bufferPool.requestBuffers(new Object());
+ } finally {
+ bufferPool.recycle(buffers, this);
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testBufferFulfilledByRecycledBuffers() throws Exception {
+ int numRequestThreads = 2;
+ AtomicReference<Throwable> exception = new AtomicReference<>();
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ Map<Object, List<MemorySegment>> buffers = new ConcurrentHashMap<>();
+
+ try {
+ Object[] owners = new Object[] {new Object(), new Object(), new Object(), new Object()};
+ for (int i = 0; i < 4; ++i) {
+ buffers.put(owners[i], bufferPool.requestBuffers(owners[i]));
+ }
+ assertEquals(0, bufferPool.getAvailableBuffers());
+
+ Thread[] requestThreads = new Thread[numRequestThreads];
+ for (int i = 0; i < numRequestThreads; ++i) {
+ requestThreads[i] =
+ new Thread(
+ () -> {
+ try {
+ Object owner = new Object();
+ buffers.put(owner, bufferPool.requestBuffers(owner));
+ } catch (Throwable throwable) {
+ exception.set(throwable);
+ }
+ });
+ requestThreads[i].start();
+ }
+
+ // recycle one by one
+ for (MemorySegment segment : buffers.remove(owners[0])) {
+ bufferPool.recycle(segment, owners[0]);
+ }
+
+ // bulk recycle
+ bufferPool.recycle(buffers.remove(owners[1]), owners[1]);
+
+ for (Thread requestThread : requestThreads) {
+ requestThread.join();
+ }
+
+ assertNull(exception.get());
+ assertEquals(0, bufferPool.getAvailableBuffers());
+ assertEquals(4, buffers.size());
+ } finally {
+ for (Object owner : buffers.keySet()) {
+ bufferPool.recycle(buffers.remove(owner), owner);
+ }
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testMultipleThreadRequestAndRecycle() throws Exception {
+ int numRequestThreads = 10;
+ AtomicReference<Throwable> exception = new AtomicReference<>();
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+
+ try {
+ Thread[] requestThreads = new Thread[numRequestThreads];
+ for (int i = 0; i < numRequestThreads; ++i) {
+ requestThreads[i] =
+ new Thread(
+ () -> {
+ try {
+ Object owner = new Object();
+ for (int j = 0; j < 100; ++j) {
+ List<MemorySegment> buffers =
+ bufferPool.requestBuffers(owner);
+ Thread.sleep(10);
+ if (j % 2 == 0) {
+ bufferPool.recycle(buffers, owner);
+ } else {
+ for (MemorySegment segment : buffers) {
+ bufferPool.recycle(segment, owner);
+ }
+ }
+ }
+ } catch (Throwable throwable) {
+ exception.set(throwable);
+ }
+ });
+ requestThreads[i].start();
+ }
+
+ for (Thread requestThread : requestThreads) {
+ requestThread.join();
+ }
+
+ assertNull(exception.get());
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ } finally {
+ bufferPool.destroy();
+ }
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testDestroy() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ bufferPool.initialize();
+
+ assertFalse(bufferPool.isDestroyed());
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+
+ List<MemorySegment> buffers = bufferPool.requestBuffers(this);
+ assertEquals(
+ bufferPool.getNumTotalBuffers() - buffers.size(), bufferPool.getAvailableBuffers());
+
+ bufferPool.destroy();
+ assertTrue(bufferPool.isDestroyed());
+ assertEquals(0, bufferPool.getAvailableBuffers());
+
+ bufferPool.recycle(buffers, this);
Review comment:
maybe add a separate test `testRecycleAfterPoolDestroyed` to test that recycled buffers will be directly discarded if the pool is destroyed?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPoolTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Copyright 2012 The Netty Project
+ * Copy from netty 4.1.32.Final
+ *
+ * The Netty Project licenses this file to you under the Apache License,
+ * version 2.0 (the "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at:
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.core.memory.MemorySegment;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BatchShuffleReadBufferPool}. */
+public class BatchShuffleReadBufferPoolTest {
+
+ @Rule public Timeout timeout = new Timeout(60, TimeUnit.SECONDS);
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalTotalBytes() {
+ createBufferPool(0, 1024);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalBufferSize() {
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, 0);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalRequestTimeout() {
+ createBufferPool(0);
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testInitializeAfterDestroy() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ bufferPool.destroy();
+ bufferPool.initialize();
+ }
+
+ @Test
+ public void testLargeTotalBytes() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(Long.MAX_VALUE, 1024);
+ assertEquals(Integer.MAX_VALUE, bufferPool.getNumTotalBuffers());
+ bufferPool.destroy();
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testTotalBytesSmallerThanBufferSize() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(4096, 32 * 1024);
+ try {
+ bufferPool.initialize();
+ } finally {
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testBufferCalculation() {
+ long totalBytes = BatchShuffleReadBufferPool.MIN_TOTAL_BYTES;
+ for (int bufferSize = 4 * 1024; bufferSize <= totalBytes; bufferSize += 1024) {
+ BatchShuffleReadBufferPool bufferPool =
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, bufferSize);
+
+ assertEquals(totalBytes, bufferPool.getTotalBytes());
+ assertEquals(totalBytes / bufferSize, bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() <= bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() > 0);
+ }
+ }
+
+ @Test
+ public void testRequestBuffers() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ List<MemorySegment> buffers = new ArrayList<>();
+
+ try {
+ buffers.addAll(bufferPool.requestBuffers(this));
+ assertEquals(bufferPool.getNumBuffersPerRequest(), buffers.size());
+ } finally {
+ bufferPool.recycle(buffers, this);
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
Review comment:
this check looks irrelevant to this test case. Can we remove it and add a separate test case `testRecycle` to test the normal simple cycle case, because currently I can only see some complex recycle test cases?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPool.java
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.TaskManagerOptions;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.TimeoutException;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A fixed-size {@link MemorySegment} pool used by batch shuffle for shuffle data read (currently
+ * only used by sort-merge blocking shuffle).
+ */
+@Internal
+public class BatchShuffleReadBufferPool {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchShuffleReadBufferPool.class);
+
+ /** Minimum total memory size in bytes of this buffer pool. */
+ public static final int MIN_TOTAL_BYTES = 32 * 1024 * 1024;
+
+ /**
+ * Memory size in bytes can be allocated from this buffer pool for a single request (8M is for
+ * better sequential read).
+ */
+ public static final int NUM_BYTES_PER_REQUEST = 8 * 1024 * 1024;
+
+ /** Total direct memory size in bytes can can be allocated and used by this buffer pool. */
+ private final long totalBytes;
+
+ /**
+ * Maximum time to wait in milliseconds when requesting read buffers from this buffer pool
+ * before throwing an exception.
+ */
+ private final long requestTimeout;
+
+ /** The number of total buffers in this buffer pool. */
+ private final int numTotalBuffers;
+
+ /** Size of each buffer in bytes in this buffer pool. */
+ private final int bufferSize;
+
+ /** The number of buffers to be returned for a single request. */
+ private final int numBuffersPerRequest;
+
+ /**
+ * The maximum number of buffers can be allocated from this buffer pool for a single buffer
+ * requester.
+ */
+ private final int maxBuffersPerRequester;
+
+ /** All available buffers in this buffer pool currently. */
+ @GuardedBy("buffers")
+ private final Queue<MemorySegment> buffers = new ArrayDeque<>();
+
+ /** Account for all the buffers requested per requester. */
+ @GuardedBy("buffers")
+ private final Map<Object, Counter> numBuffersAllocated = new HashMap<>();
+
+ /** Whether this buffer pool has been destroyed or not. */
+ @GuardedBy("buffers")
+ private boolean destroyed;
+
+ /** Whether this buffer pool has been initialized or not. */
+ @GuardedBy("buffers")
+ private boolean initialized;
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize) {
+ // 5 min default buffer request timeout
+ this(totalBytes, bufferSize, 5 * 60 * 1000);
+ }
+
+ public BatchShuffleReadBufferPool(long totalBytes, int bufferSize, long requestTimeout) {
+ checkArgument(totalBytes > 0, "Total memory size must be positive.");
+ checkArgument(bufferSize > 0, "Size of buffer must be positive.");
+ checkArgument(requestTimeout > 0, "Request timeout must be positive.");
+
+ this.totalBytes = totalBytes;
+ this.bufferSize = bufferSize;
+ this.requestTimeout = requestTimeout;
+
+ this.numTotalBuffers = (int) Math.min(totalBytes / bufferSize, Integer.MAX_VALUE);
+ this.numBuffersPerRequest =
+ Math.min(numTotalBuffers, Math.max(1, NUM_BYTES_PER_REQUEST / bufferSize));
+ this.maxBuffersPerRequester =
+ Math.max(
+ 4 * numBuffersPerRequest,
+ numTotalBuffers / BatchShuffleReadIOExecutor.MAX_NUM_THREADS);
+ }
+
+ public long getTotalBytes() {
+ return totalBytes;
+ }
+
+ public long getRequestTimeout() {
+ return requestTimeout;
+ }
+
+ public int getNumBuffersPerRequest() {
+ return numBuffersPerRequest;
+ }
+
+ public int getMaxBuffersPerRequester() {
+ return maxBuffersPerRequester;
+ }
+
+ public int getBufferSize() {
+ return bufferSize;
+ }
+
+ public int getNumTotalBuffers() {
+ return numTotalBuffers;
+ }
+
+ public int getAvailableBuffers() {
+ synchronized (buffers) {
+ return buffers.size();
+ }
+ }
+
+ /** Initializes this buffer pool which allocates all the buffers. */
+ public void initialize() {
+ LOG.info(
+ "Initializing batch shuffle IO buffer pool: numBuffers={}, bufferSize={}.",
+ numTotalBuffers,
+ bufferSize);
+
+ checkState(
+ totalBytes >= bufferSize,
+ String.format(
+ "Illegal configuration, config value for %s must be no smaller than %s,"
+ + " please increase %s to at least %s bytes.",
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.MEMORY_SEGMENT_SIZE.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ bufferSize));
+
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (initialized) {
+ return;
+ }
+ initialized = true;
+
+ try {
+ for (int i = 0; i < numTotalBuffers; ++i) {
+ buffers.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(bufferSize));
+ }
+ } catch (OutOfMemoryError outOfMemoryError) {
+ int allocated = buffers.size();
+ buffers.clear();
+ throw new OutOfMemoryError(
+ String.format(
+ "Can't allocate enough direct buffer for batch shuffle read buffer "
+ + "pool (bytes allocated: %d, bytes still needed: %d). To "
+ + "avoid the exception, you need to do one of the following"
+ + " adjustments: 1) If you have ever decreased %s, you need"
+ + " to undo the decrement; 2) If you ever increased %s, you"
+ + " should also increase %s; 3) If neither the above cases,"
+ + " it usually means some other parts of your application "
+ + "have consumed too many direct memory and the value of %s"
+ + " should be increased.",
+ allocated * bufferSize,
+ (numTotalBuffers - allocated) * bufferSize,
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.NETWORK_BATCH_SHUFFLE_READ_MEMORY.key(),
+ TaskManagerOptions.FRAMEWORK_OFF_HEAP_MEMORY.key(),
+ TaskManagerOptions.TASK_OFF_HEAP_MEMORY.key()));
+ }
+ }
+ }
+
+ /**
+ * Requests a collection of buffers (determined by {@link #numBuffersPerRequest}) from this
+ * buffer pool. Exception will be thrown if no enough buffers can be allocated in the given
+ * timeout and the corresponding requester is not holding any allocated buffers currently.
+ */
+ public List<MemorySegment> requestBuffers(Object owner) throws Exception {
+ checkArgument(owner != null, "Owner must be not null.");
+
+ List<MemorySegment> allocated = new ArrayList<>(numBuffersPerRequest);
+ synchronized (buffers) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ if (!initialized) {
+ initialize();
+ }
+
+ long startTime = System.currentTimeMillis();
+ Counter counter = numBuffersAllocated.get(owner);
+ while (buffers.size() < numBuffersPerRequest
+ || (counter != null
+ && counter.get() + numBuffersPerRequest > maxBuffersPerRequester)) {
+ checkState(!destroyed, "Buffer pool is already destroyed.");
+
+ buffers.wait(requestTimeout);
Review comment:
close this comment because it is outdated with the removal of `maxBuffersPerRequester`
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPoolTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Copyright 2012 The Netty Project
+ * Copy from netty 4.1.32.Final
+ *
+ * The Netty Project licenses this file to you under the Apache License,
+ * version 2.0 (the "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at:
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.core.memory.MemorySegment;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BatchShuffleReadBufferPool}. */
+public class BatchShuffleReadBufferPoolTest {
+
+ @Rule public Timeout timeout = new Timeout(60, TimeUnit.SECONDS);
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalTotalBytes() {
+ createBufferPool(0, 1024);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalBufferSize() {
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, 0);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalRequestTimeout() {
+ createBufferPool(0);
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testInitializeAfterDestroy() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ bufferPool.destroy();
+ bufferPool.initialize();
+ }
+
+ @Test
+ public void testLargeTotalBytes() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(Long.MAX_VALUE, 1024);
+ assertEquals(Integer.MAX_VALUE, bufferPool.getNumTotalBuffers());
+ bufferPool.destroy();
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testTotalBytesSmallerThanBufferSize() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(4096, 32 * 1024);
+ try {
+ bufferPool.initialize();
+ } finally {
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testBufferCalculation() {
+ long totalBytes = BatchShuffleReadBufferPool.MIN_TOTAL_BYTES;
+ for (int bufferSize = 4 * 1024; bufferSize <= totalBytes; bufferSize += 1024) {
+ BatchShuffleReadBufferPool bufferPool =
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, bufferSize);
+
+ assertEquals(totalBytes, bufferPool.getTotalBytes());
+ assertEquals(totalBytes / bufferSize, bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() <= bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() > 0);
+ }
+ }
+
+ @Test
+ public void testRequestBuffers() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ List<MemorySegment> buffers = new ArrayList<>();
+
+ try {
+ buffers.addAll(bufferPool.requestBuffers(this));
+ assertEquals(bufferPool.getNumBuffersPerRequest(), buffers.size());
+ } finally {
+ bufferPool.recycle(buffers, this);
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ bufferPool.destroy();
+ }
+ }
+
+ @Test(expected = TimeoutException.class)
+ public void testRequestBuffersTimeout() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(1000);
+ List<MemorySegment> buffers = new ArrayList<>();
+
+ try {
+ for (int i = 0; i < 4; ++i) {
+ buffers.addAll(bufferPool.requestBuffers(this));
+ }
+ bufferPool.requestBuffers(new Object());
+ } finally {
+ bufferPool.recycle(buffers, this);
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
Review comment:
same as the above comment. Let's have a separate test case for this check.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/io/disk/BatchShuffleReadBufferPoolTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Copyright 2012 The Netty Project
+ * Copy from netty 4.1.32.Final
+ *
+ * The Netty Project licenses this file to you under the Apache License,
+ * version 2.0 (the "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at:
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.runtime.io.disk;
+
+import org.apache.flink.core.memory.MemorySegment;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.Timeout;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BatchShuffleReadBufferPool}. */
+public class BatchShuffleReadBufferPoolTest {
+
+ @Rule public Timeout timeout = new Timeout(60, TimeUnit.SECONDS);
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalTotalBytes() {
+ createBufferPool(0, 1024);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalBufferSize() {
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, 0);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testIllegalRequestTimeout() {
+ createBufferPool(0);
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testInitializeAfterDestroy() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ bufferPool.destroy();
+ bufferPool.initialize();
+ }
+
+ @Test
+ public void testLargeTotalBytes() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(Long.MAX_VALUE, 1024);
+ assertEquals(Integer.MAX_VALUE, bufferPool.getNumTotalBuffers());
+ bufferPool.destroy();
+ }
+
+ @Test(expected = IllegalStateException.class)
+ public void testTotalBytesSmallerThanBufferSize() {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(4096, 32 * 1024);
+ try {
+ bufferPool.initialize();
+ } finally {
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testBufferCalculation() {
+ long totalBytes = BatchShuffleReadBufferPool.MIN_TOTAL_BYTES;
+ for (int bufferSize = 4 * 1024; bufferSize <= totalBytes; bufferSize += 1024) {
+ BatchShuffleReadBufferPool bufferPool =
+ createBufferPool(BatchShuffleReadBufferPool.MIN_TOTAL_BYTES, bufferSize);
+
+ assertEquals(totalBytes, bufferPool.getTotalBytes());
+ assertEquals(totalBytes / bufferSize, bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() <= bufferPool.getNumTotalBuffers());
+ assertTrue(bufferPool.getNumBuffersPerRequest() > 0);
+ }
+ }
+
+ @Test
+ public void testRequestBuffers() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ List<MemorySegment> buffers = new ArrayList<>();
+
+ try {
+ buffers.addAll(bufferPool.requestBuffers(this));
+ assertEquals(bufferPool.getNumBuffersPerRequest(), buffers.size());
+ } finally {
+ bufferPool.recycle(buffers, this);
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ bufferPool.destroy();
+ }
+ }
+
+ @Test(expected = TimeoutException.class)
+ public void testRequestBuffersTimeout() throws Exception {
+ BatchShuffleReadBufferPool bufferPool = createBufferPool(1000);
+ List<MemorySegment> buffers = new ArrayList<>();
+
+ try {
+ for (int i = 0; i < 4; ++i) {
+ buffers.addAll(bufferPool.requestBuffers(this));
+ }
+ bufferPool.requestBuffers(new Object());
+ } finally {
+ bufferPool.recycle(buffers, this);
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testBufferFulfilledByRecycledBuffers() throws Exception {
+ int numRequestThreads = 2;
+ AtomicReference<Throwable> exception = new AtomicReference<>();
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+ Map<Object, List<MemorySegment>> buffers = new ConcurrentHashMap<>();
+
+ try {
+ Object[] owners = new Object[] {new Object(), new Object(), new Object(), new Object()};
+ for (int i = 0; i < 4; ++i) {
+ buffers.put(owners[i], bufferPool.requestBuffers(owners[i]));
+ }
+ assertEquals(0, bufferPool.getAvailableBuffers());
+
+ Thread[] requestThreads = new Thread[numRequestThreads];
+ for (int i = 0; i < numRequestThreads; ++i) {
+ requestThreads[i] =
+ new Thread(
+ () -> {
+ try {
+ Object owner = new Object();
+ buffers.put(owner, bufferPool.requestBuffers(owner));
+ } catch (Throwable throwable) {
+ exception.set(throwable);
+ }
+ });
+ requestThreads[i].start();
+ }
+
+ // recycle one by one
+ for (MemorySegment segment : buffers.remove(owners[0])) {
+ bufferPool.recycle(segment, owners[0]);
+ }
+
+ // bulk recycle
+ bufferPool.recycle(buffers.remove(owners[1]), owners[1]);
+
+ for (Thread requestThread : requestThreads) {
+ requestThread.join();
+ }
+
+ assertNull(exception.get());
+ assertEquals(0, bufferPool.getAvailableBuffers());
+ assertEquals(4, buffers.size());
+ } finally {
+ for (Object owner : buffers.keySet()) {
+ bufferPool.recycle(buffers.remove(owner), owner);
+ }
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ bufferPool.destroy();
+ }
+ }
+
+ @Test
+ public void testMultipleThreadRequestAndRecycle() throws Exception {
+ int numRequestThreads = 10;
+ AtomicReference<Throwable> exception = new AtomicReference<>();
+ BatchShuffleReadBufferPool bufferPool = createBufferPool();
+
+ try {
+ Thread[] requestThreads = new Thread[numRequestThreads];
+ for (int i = 0; i < numRequestThreads; ++i) {
+ requestThreads[i] =
+ new Thread(
+ () -> {
+ try {
+ Object owner = new Object();
+ for (int j = 0; j < 100; ++j) {
+ List<MemorySegment> buffers =
+ bufferPool.requestBuffers(owner);
+ Thread.sleep(10);
+ if (j % 2 == 0) {
+ bufferPool.recycle(buffers, owner);
+ } else {
+ for (MemorySegment segment : buffers) {
+ bufferPool.recycle(segment, owner);
+ }
+ }
+ }
+ } catch (Throwable throwable) {
+ exception.set(throwable);
+ }
+ });
+ requestThreads[i].start();
+ }
+
+ for (Thread requestThread : requestThreads) {
+ requestThread.join();
+ }
+
+ assertNull(exception.get());
+ assertEquals(bufferPool.getNumTotalBuffers(), bufferPool.getAvailableBuffers());
+ } finally {
+ bufferPool.destroy();
+ }
+ }
+
+ @Test(expected = IllegalStateException.class)
Review comment:
could we have this test case to test normal case only and have a separate test case `testRequestBuffersAfterPoolDestroyed` to test `requestBuffers` after destroy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org