You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/07/20 16:37:40 UTC
[GitHub] [flink] reswqa commented on a diff in pull request #20320: [FLINK-28519][network] Fix the bug that SortMergeResultPartitionReadScheduler may not read data sequentially
reswqa commented on code in PR #20320:
URL: https://github.com/apache/flink/pull/20320#discussion_r925818232
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java:
##########
@@ -152,12 +160,34 @@ class SortMergeResultPartitionReadScheduler implements Runnable, BufferRecycler
@Override
public synchronized void run() {
- Queue<SortMergeSubpartitionReader> availableReaders = getAvailableReaders();
-
- Queue<MemorySegment> buffers = allocateBuffers(availableReaders);
+ Set<SortMergeSubpartitionReader> finishedReaders = new HashSet<>();
+ Queue<MemorySegment> buffers;
+ try {
+ buffers = allocateBuffers();
+ } catch (Throwable throwable) {
+ // fail all pending subpartition readers immediately if any exception occurs
+ LOG.error("Failed to request buffers for data reading.", throwable);
+ failSubpartitionReaders(getAllReaders(), throwable);
Review Comment:
If a failed reader not removed from `sortedBuffers` in time, It can only be removed after the next poll, but this failed reader will read data from disk int that case, which is an unnecessary extra overhead. IMO, we need clear `sortedReaders` when `allocateBuffers` throw exception.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java:
##########
@@ -283,13 +284,26 @@ private void mayNotifyReleased() {
}
}
- private Queue<SortMergeSubpartitionReader> getAvailableReaders() {
+ private Queue<SortMergeSubpartitionReader> getAllReaders() {
synchronized (lock) {
if (isReleased) {
return new ArrayDeque<>();
}
+ return new ArrayDeque<>(allReaders);
+ }
+ }
- return new PriorityQueue<>(allReaders);
+ @Nullable
+ private SortMergeSubpartitionReader getNextReaderToRead(
Review Comment:
```suggestion
private SortMergeSubpartitionReader addPreviousAndGetNextReader(
```
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java:
##########
@@ -152,12 +160,34 @@ class SortMergeResultPartitionReadScheduler implements Runnable, BufferRecycler
@Override
public synchronized void run() {
- Queue<SortMergeSubpartitionReader> availableReaders = getAvailableReaders();
-
- Queue<MemorySegment> buffers = allocateBuffers(availableReaders);
+ Set<SortMergeSubpartitionReader> finishedReaders = new HashSet<>();
+ Queue<MemorySegment> buffers;
+ try {
+ buffers = allocateBuffers();
+ } catch (Throwable throwable) {
+ // fail all pending subpartition readers immediately if any exception occurs
+ LOG.error("Failed to request buffers for data reading.", throwable);
+ failSubpartitionReaders(getAllReaders(), throwable);
Review Comment:
`release` and `releaseSubpartitionReader` should take care of `sortedReaders ` also. These two method will add reader to `failedReaders` set, I suggest add check to `getNextReaderToRead`, when the polled reader contains in `failedReaders`, we can ignore it and poll next until meet the reader not failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org