You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "WencongLiu (via GitHub)" <gi...@apache.org> on 2023/06/25 04:31:04 UTC
[GitHub] [flink] WencongLiu opened a new pull request, #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
WencongLiu opened a new pull request, #22855:
URL: https://github.com/apache/flink/pull/22855
## What is the purpose of the change
### Introduce the RemoteTierConsumerAgent
1. Introduce the HashPartitionFileReader
2. Introduce the RemoteTierConsumerAgent and RemoteStorageFileScanner
3. Introduce the RemoteTierConsumerAgent to TieredStorageConsumerClient
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a diff in pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #22855:
URL: https://github.com/apache/flink/pull/22855#discussion_r1257405998
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskTierConsumerAgent.java:
##########
@@ -42,6 +44,8 @@ public class DiskTierConsumerAgent implements TierConsumerAgent {
Map<TieredStorageSubpartitionId, CompletableFuture<NettyConnectionReader>>>
nettyConnectionReaders = new HashMap<>();
+ private AvailabilityAndPriorityNotifier notifier;
Review Comment:
This is `null` initially. We should mark it `@Nullable`, and add checks before using it.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityAndPriorityNotifier;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to watch the required segments and scan
+ * the existence status of the segments. If the segment file is found, it will notify the
+ * availability of segment file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 100ms. */
+ private static final int INITIAL_SCAN_INTERVAL_MS = 100;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL_MS = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /** The key is partition id and subpartition id, the value is required segment id. */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private FileSystem remoteFileSystem;
+
+ private AvailabilityAndPriorityNotifier notifier;
+
+ private int attemptNumber = 0;
+
+ public RemoteStorageScanner(String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.requiredSegmentIds = new ConcurrentHashMap<>();
+ this.scannedMaxSegmentIds = new ConcurrentHashMap<>();
+ this.scanStrategy = new ScanStrategy(INITIAL_SCAN_INTERVAL_MS, MAX_SCAN_INTERVAL_MS);
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.schedule(
+ this, scanStrategy.getInterval(attemptNumber), TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Watch the segment for a specific subpartition in the {@link RemoteStorageScanner}.
+ *
+ * <p>If a segment with a larger or equal id already exists, the current segment won't be
+ * watched.
+ *
+ * <p>If a segment with a smaller segment id is still being watched, the current segment will
+ * replace it because the smaller segment should have been consumed. This method ensures that
+ * only one segment file can be watched for each subpartition.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void watchSegment(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId> key =
+ Tuple2.of(partitionId, subpartitionId);
+ if (scannedMaxSegmentIds.getOrDefault(key, -1) >= segmentId) {
+ return;
+ }
+ requiredSegmentIds.put(key, segmentId);
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the watched segment ids and check related file status. */
+ @Override
+ public void run() {
+ Iterator<Map.Entry<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>>
+ iterator = requiredSegmentIds.entrySet().iterator();
+ boolean scanned = false;
+ while (iterator.hasNext()) {
+ Map.Entry<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer> ids =
+ iterator.next();
+ TieredStoragePartitionId partitionId = ids.getKey().f0;
+ TieredStorageSubpartitionId subpartitionId = ids.getKey().f1;
+ int requiredSegmentId = ids.getValue();
+ int maxSegmentId = scannedMaxSegmentIds.getOrDefault(ids.getKey(), -1);
+ if (maxSegmentId >= requiredSegmentId) {
+ scanned = true;
+ iterator.remove();
+ notifier.notifyAvailableAndPriority(partitionId, subpartitionId, false);
+ } else {
+ // The segment should be watched again because it's not found.
+ // If the segment belongs to other tiers and has been consumed, the segment will be
+ // replaced by newly watched segment with larger segment id. This logic is ensured
+ // by the method {@code watchSegment}.
+ scanMaxSegmentId(partitionId, subpartitionId);
+ }
+ }
+ attemptNumber = scanned ? 0 : attemptNumber + 1;
+ scannerExecutor.schedule(
+ this, scanStrategy.getInterval(attemptNumber), TimeUnit.MILLISECONDS);
+ }
+
+ public void registerAvailabilityAndPriorityNotifier(AvailabilityAndPriorityNotifier retriever) {
+ this.notifier = retriever;
+ }
+
+ // ------------------------------------------------------------------------
+ // Internal Methods
+ // ------------------------------------------------------------------------
+
+ /**
+ * Scan the max segment id of segment files for the specific partition and subpartition. The max
+ * segment id can be obtained from a file named by max segment id.
+ *
+ * @param partitionId the partition id.
+ * @param subpartitionId the subpartition id.
+ */
+ private void scanMaxSegmentId(
+ TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId) {
+ Path segmentFinishDir =
+ getSegmentFinishDirPath(
+ baseRemoteStoragePath, partitionId, subpartitionId.getSubpartitionId());
+ FileStatus[] fileStatuses;
+ try {
+ if (!remoteFileSystem.exists(segmentFinishDir)) {
+ return;
+ }
+ fileStatuses = remoteFileSystem.listStatus(segmentFinishDir);
+ } catch (IOException e) {
+ throw new RuntimeException(
+ "Failed to list the segment finish file. " + segmentFinishDir, e);
+ }
+ if (fileStatuses.length == 0) {
+ return;
+ }
+ checkState(fileStatuses.length == 1);
+ scannedMaxSegmentIds.put(
+ Tuple2.of(partitionId, subpartitionId),
+ Integer.parseInt(fileStatuses[0].getPath().getName()));
+ }
+
+ /**
+ * The strategy is used to decide the scan interval of {@link RemoteStorageScanner}. The
+ * interval will be updated exponentially and restricted by max value.
+ */
+ private static class ScanStrategy {
+
+ private final int initialScanInterval;
+
+ private final int maxScanInterval;
+
+ private ScanStrategy(int initialScanInterval, int maxScanInterval) {
+ checkArgument(
+ initialScanInterval > 0,
+ "initialScanInterval must be positive, was %s",
+ initialScanInterval);
+ checkArgument(
+ maxScanInterval > 0,
+ "maxScanInterval must be positive, was %s",
+ maxScanInterval);
+ checkArgument(
+ initialScanInterval <= maxScanInterval,
+ "initialScanInterval must be lower than or equal to maxScanInterval",
+ maxScanInterval);
+ this.initialScanInterval = initialScanInterval;
+ this.maxScanInterval = maxScanInterval;
+ }
+
+ private long getInterval(int attempt) {
+ checkArgument(attempt >= 0, "attempt must not be negative (%s)", attempt);
+ final long interval = initialScanInterval * Math.round(Math.pow(2, attempt));
+ return interval >= 0 && interval < maxScanInterval ? interval : maxScanInterval;
+ }
Review Comment:
Relying on `interval >= 0` to check for overflows can be problematic, because the `attempt` is keep growing and it's possible that an overflowed interval may still fall into the range `[0, maxScanInterval)`. Alternatively, you may replace `attempt` with a `lastInterval`, which stops growing after reaching the max.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityAndPriorityNotifier;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to watch the required segments and scan
+ * the existence status of the segments. If the segment file is found, it will notify the
+ * availability of segment file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 100ms. */
+ private static final int INITIAL_SCAN_INTERVAL_MS = 100;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL_MS = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /** The key is partition id and subpartition id, the value is required segment id. */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private FileSystem remoteFileSystem;
+
+ private AvailabilityAndPriorityNotifier notifier;
+
+ private int attemptNumber = 0;
+
+ public RemoteStorageScanner(String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.requiredSegmentIds = new ConcurrentHashMap<>();
+ this.scannedMaxSegmentIds = new ConcurrentHashMap<>();
+ this.scanStrategy = new ScanStrategy(INITIAL_SCAN_INTERVAL_MS, MAX_SCAN_INTERVAL_MS);
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.schedule(
+ this, scanStrategy.getInterval(attemptNumber), TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Watch the segment for a specific subpartition in the {@link RemoteStorageScanner}.
+ *
+ * <p>If a segment with a larger or equal id already exists, the current segment won't be
+ * watched.
+ *
+ * <p>If a segment with a smaller segment id is still being watched, the current segment will
+ * replace it because the smaller segment should have been consumed. This method ensures that
+ * only one segment file can be watched for each subpartition.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void watchSegment(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId> key =
+ Tuple2.of(partitionId, subpartitionId);
+ if (scannedMaxSegmentIds.getOrDefault(key, -1) >= segmentId) {
+ return;
+ }
+ requiredSegmentIds.put(key, segmentId);
Review Comment:
Minor: May use `Map#compute` to reduce map access.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskTierConsumerAgent.java:
##########
@@ -42,6 +44,8 @@ public class DiskTierConsumerAgent implements TierConsumerAgent {
Map<TieredStorageSubpartitionId, CompletableFuture<NettyConnectionReader>>>
nettyConnectionReaders = new HashMap<>();
+ private AvailabilityAndPriorityNotifier notifier;
Review Comment:
And same for `MemoryTierConsumerAgent`.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteTierConsumerAgent.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.PartitionFileReader;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityAndPriorityNotifier;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.TierConsumerAgent;
+import org.apache.flink.util.ExceptionUtils;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/** The data client is used to fetch data from remote tier. */
+public class RemoteTierConsumerAgent implements TierConsumerAgent {
+
+ private final RemoteStorageScanner remoteStorageScanner;
+
+ private final PartitionFileReader partitionFileReader;
+
+ /**
+ * The current reading buffer indexes and segment ids stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is buffer index and segment id.
+ */
+ private final Map<
+ TieredStoragePartitionId,
+ Map<TieredStorageSubpartitionId, Tuple2<Integer, Integer>>>
+ currentBufferIndexAndSegmentIds;
+
+ private final int bufferSizeBytes;
+
+ private AvailabilityAndPriorityNotifier notifier;
Review Comment:
Same here for the potential null-value.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java:
##########
@@ -1146,21 +1151,34 @@ private void setupTieredStorageNettyService(
int channelIndex = index;
channelSuppliers.add(() -> channels[channelIndex]);
}
- nettyService.setupInputChannels(
- tieredStorageConsumerSpecs,
- channelSuppliers,
- new NettyConnectionReaderAvailabilityAndPriorityHelper() {
- @Override
- public void notifyReaderAvailableAndPriority(
- int channelIndex, boolean isPriority) {
- queueChannel(channels[channelIndex], null, isPriority);
- }
+ nettyService.setupInputChannels(tieredStorageConsumerSpecs, channelSuppliers);
+ }
- @Override
- public void updatePrioritySequenceNumber(int channelIndex, int sequenceNumber) {
- lastPrioritySequenceNumber[channelIndex] = sequenceNumber;
- }
- });
+ /** The default implementation of {@link AvailabilityAndPriorityNotifier}. */
+ private class AvailabilityAndPriorityNotifierImpl implements AvailabilityAndPriorityNotifier {
+
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ channelIndexes;
+
+ public AvailabilityAndPriorityNotifierImpl() {
+ this.channelIndexes = new HashMap<>();
+ for (int index = 0; index < checkNotNull(tieredStorageConsumerSpecs).size(); index++) {
+ TieredStorageConsumerSpec spec = tieredStorageConsumerSpecs.get(index);
+ channelIndexes
+ .computeIfAbsent(spec.getPartitionId(), ignore -> new HashMap<>())
+ .put(spec.getSubpartitionId(), index);
+ }
+ }
+
+ public void notifyAvailableAndPriority(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ boolean isPriority) {
+ queueChannel(
+ channels[channelIndexes.get(partitionId).get(subpartitionId)],
+ null,
+ isPriority);
+ }
Review Comment:
Why don't we need the last priority sequence number anymore?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityAndPriorityNotifier;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to watch the required segments and scan
+ * the existence status of the segments. If the segment file is found, it will notify the
+ * availability of segment file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 100ms. */
+ private static final int INITIAL_SCAN_INTERVAL_MS = 100;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL_MS = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /** The key is partition id and subpartition id, the value is required segment id. */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private FileSystem remoteFileSystem;
Review Comment:
Why is this not `final`?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteTierFactory.java:
##########
@@ -90,7 +91,12 @@ public TierProducerAgent createProducerAgent(
public TierConsumerAgent createConsumerAgent(
List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
TieredStorageNettyService nettyService) {
- // TODO, implement the remote tier consumer agent
- return null;
+ PartitionFileReader partitionFileReader =
+ SegmentPartitionFile.createPartitionFileReader(
+ getTieredStoragePath(remoteStorageBasePath));
+ return new RemoteTierConsumerAgent(
+ new RemoteStorageScanner(getTieredStoragePath(remoteStorageBasePath)),
+ partitionFileReader,
+ bufferSizeBytes);
Review Comment:
Minor:
1. `getTieredStoragePath(remoteStorageBasePath)` can be deduplicated
2. It might be better to align code-style for `partitionFileReader` and `remoteStorageScanner`, i.e. creating them either both inline when creating the agent, or both before creating the agent.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityAndPriorityNotifier;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to watch the required segments and scan
+ * the existence status of the segments. If the segment file is found, it will notify the
+ * availability of segment file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 100ms. */
+ private static final int INITIAL_SCAN_INTERVAL_MS = 100;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL_MS = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /** The key is partition id and subpartition id, the value is required segment id. */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private FileSystem remoteFileSystem;
+
+ private AvailabilityAndPriorityNotifier notifier;
+
+ private int attemptNumber = 0;
+
+ public RemoteStorageScanner(String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.requiredSegmentIds = new ConcurrentHashMap<>();
+ this.scannedMaxSegmentIds = new ConcurrentHashMap<>();
+ this.scanStrategy = new ScanStrategy(INITIAL_SCAN_INTERVAL_MS, MAX_SCAN_INTERVAL_MS);
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.schedule(
+ this, scanStrategy.getInterval(attemptNumber), TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Watch the segment for a specific subpartition in the {@link RemoteStorageScanner}.
+ *
+ * <p>If a segment with a larger or equal id already exists, the current segment won't be
+ * watched.
+ *
+ * <p>If a segment with a smaller segment id is still being watched, the current segment will
+ * replace it because the smaller segment should have been consumed. This method ensures that
+ * only one segment file can be watched for each subpartition.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void watchSegment(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId> key =
+ Tuple2.of(partitionId, subpartitionId);
+ if (scannedMaxSegmentIds.getOrDefault(key, -1) >= segmentId) {
+ return;
+ }
+ requiredSegmentIds.put(key, segmentId);
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the watched segment ids and check related file status. */
+ @Override
+ public void run() {
+ Iterator<Map.Entry<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>>
+ iterator = requiredSegmentIds.entrySet().iterator();
+ boolean scanned = false;
+ while (iterator.hasNext()) {
+ Map.Entry<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer> ids =
+ iterator.next();
+ TieredStoragePartitionId partitionId = ids.getKey().f0;
+ TieredStorageSubpartitionId subpartitionId = ids.getKey().f1;
+ int requiredSegmentId = ids.getValue();
+ int maxSegmentId = scannedMaxSegmentIds.getOrDefault(ids.getKey(), -1);
+ if (maxSegmentId >= requiredSegmentId) {
+ scanned = true;
+ iterator.remove();
+ notifier.notifyAvailableAndPriority(partitionId, subpartitionId, false);
Review Comment:
This doesn't seem right. A required segment may not exist in remote tier at all. `maxSegmentId >= requiredSegmentId` does not necessarily mean the segment exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a diff in pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #22855:
URL: https://github.com/apache/flink/pull/22855#discussion_r1248809827
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageIdMappingUtils;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageConsumerSpec;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSegmentFinishPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSubpartitionPath;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /**
+ * Tuple3 containing partition id and subpartition id and segment id. The tuple3 is stored in
+ * queue and indicates the required segment file.
+ */
+ private final Queue<Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>>
+ requiredSegmentIds;
+
+ /**
+ * The channel indexes stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is related channel index.
+ */
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ channelIndexes;
+
+ private final String baseRemoteStoragePath;
+
+ private FileSystem remoteFileSystem;
+
+ private RemoteStorageScannerAvailabilityAndPriorityHelper helper;
+
+ public RemoteStorageScanner(
+ List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
+ String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.channelIndexes = new HashMap<>();
+ for (int index = 0; index < tieredStorageConsumerSpecs.size(); index++) {
+ TieredStorageConsumerSpec spec = tieredStorageConsumerSpecs.get(index);
+ channelIndexes
+ .computeIfAbsent(spec.getPartitionId(), ignore -> new HashMap<>())
+ .put(spec.getSubpartitionId(), index);
+ }
+ this.requiredSegmentIds = new LinkedBlockingDeque<>();
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.scheduleAtFixedRate(this, 0, 10, TimeUnit.MILLISECONDS);
Review Comment:
1. 10ms might be too aggressive. This would likely overwhelm the remote storage. Moreover, as we already fallback to the remote tier, I don't think it's that critical to discover the segments very timely.
2. We should at least make this constant a static final field.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScannerAvailabilityAndPriorityHelper.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+/**
+ * {@link RemoteStorageScannerAvailabilityAndPriorityHelper} is used to help the {@link
+ * RemoteStorageScanner} to notify the availability and priority of the reading process for the
+ * specific partition and subpartition on remote storage.
+ */
+public interface RemoteStorageScannerAvailabilityAndPriorityHelper {
Review Comment:
Looks like this is identical to `NettyConnectionReaderAvailabilityAndPriorityHelper`, and even their implementations `SingleInputGate` are identical. They should be deduplicated
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageIdMappingUtils;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageConsumerSpec;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSegmentFinishPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSubpartitionPath;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /**
+ * Tuple3 containing partition id and subpartition id and segment id. The tuple3 is stored in
+ * queue and indicates the required segment file.
+ */
+ private final Queue<Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>>
+ requiredSegmentIds;
+
+ /**
+ * The channel indexes stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is related channel index.
+ */
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ channelIndexes;
+
+ private final String baseRemoteStoragePath;
+
+ private FileSystem remoteFileSystem;
+
+ private RemoteStorageScannerAvailabilityAndPriorityHelper helper;
+
+ public RemoteStorageScanner(
+ List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
+ String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.channelIndexes = new HashMap<>();
+ for (int index = 0; index < tieredStorageConsumerSpecs.size(); index++) {
+ TieredStorageConsumerSpec spec = tieredStorageConsumerSpecs.get(index);
+ channelIndexes
+ .computeIfAbsent(spec.getPartitionId(), ignore -> new HashMap<>())
+ .put(spec.getSubpartitionId(), index);
+ }
+ this.requiredSegmentIds = new LinkedBlockingDeque<>();
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.scheduleAtFixedRate(this, 0, 10, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Register a segment id to the {@link RemoteStorageScanner}. If the scanner discovers the
+ * segment file exists, it will trigger the reading process of the segment file.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void registerSegmentId(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ requiredSegmentIds.add(Tuple3.of(partitionId, subpartitionId, segmentId));
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the registered segment ids and check related file status. */
+ @Override
+ public void run() {
+ int segmentFileNumber = requiredSegmentIds.size();
+ while (segmentFileNumber-- > 0) {
+ Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>
+ partitionIdAndSubpartitionIdAndSegmentId =
+ checkNotNull(requiredSegmentIds.poll());
+ if (checkFileExist(
+ partitionIdAndSubpartitionIdAndSegmentId.f0,
+ partitionIdAndSubpartitionIdAndSegmentId.f1,
+ partitionIdAndSubpartitionIdAndSegmentId.f2)) {
+ helper.notifyAvailableAndPriority(
+ channelIndexes
+ .get(partitionIdAndSubpartitionIdAndSegmentId.f0)
+ .get(partitionIdAndSubpartitionIdAndSegmentId.f1),
+ false);
+ } else {
+ requiredSegmentIds.add(partitionIdAndSubpartitionIdAndSegmentId);
+ }
+ }
+ }
+
+ public void setupRemoteStorageScannerAvailabilityAndPriorityHelper(
+ RemoteStorageScannerAvailabilityAndPriorityHelper helper) {
+ this.helper = helper;
+ }
+
+ public void triggerNextRoundReading(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ boolean isPriority) {
+ helper.notifyAvailableAndPriority(
+ channelIndexes.get(partitionId).get(subpartitionId), isPriority);
+ }
+
+ public void updatePrioritySequenceNumber(
Review Comment:
Same here. Why updating the priority and sequence number through the scanner?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageIdMappingUtils;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageConsumerSpec;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSegmentFinishPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSubpartitionPath;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /**
+ * Tuple3 containing partition id and subpartition id and segment id. The tuple3 is stored in
+ * queue and indicates the required segment file.
+ */
+ private final Queue<Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>>
+ requiredSegmentIds;
+
+ /**
+ * The channel indexes stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is related channel index.
+ */
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ channelIndexes;
+
+ private final String baseRemoteStoragePath;
+
+ private FileSystem remoteFileSystem;
+
+ private RemoteStorageScannerAvailabilityAndPriorityHelper helper;
+
+ public RemoteStorageScanner(
+ List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
+ String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.channelIndexes = new HashMap<>();
+ for (int index = 0; index < tieredStorageConsumerSpecs.size(); index++) {
+ TieredStorageConsumerSpec spec = tieredStorageConsumerSpecs.get(index);
+ channelIndexes
+ .computeIfAbsent(spec.getPartitionId(), ignore -> new HashMap<>())
+ .put(spec.getSubpartitionId(), index);
+ }
+ this.requiredSegmentIds = new LinkedBlockingDeque<>();
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.scheduleAtFixedRate(this, 0, 10, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Register a segment id to the {@link RemoteStorageScanner}. If the scanner discovers the
+ * segment file exists, it will trigger the reading process of the segment file.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void registerSegmentId(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ requiredSegmentIds.add(Tuple3.of(partitionId, subpartitionId, segmentId));
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the registered segment ids and check related file status. */
+ @Override
+ public void run() {
+ int segmentFileNumber = requiredSegmentIds.size();
+ while (segmentFileNumber-- > 0) {
+ Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>
+ partitionIdAndSubpartitionIdAndSegmentId =
+ checkNotNull(requiredSegmentIds.poll());
+ if (checkFileExist(
+ partitionIdAndSubpartitionIdAndSegmentId.f0,
+ partitionIdAndSubpartitionIdAndSegmentId.f1,
+ partitionIdAndSubpartitionIdAndSegmentId.f2)) {
+ helper.notifyAvailableAndPriority(
+ channelIndexes
+ .get(partitionIdAndSubpartitionIdAndSegmentId.f0)
+ .get(partitionIdAndSubpartitionIdAndSegmentId.f1),
+ false);
+ } else {
+ requiredSegmentIds.add(partitionIdAndSubpartitionIdAndSegmentId);
Review Comment:
What if a segment is no longer needed? How do we remove it? E.g., the consumer doesn't know which tier a segment is in and asks for each tier, and later obtained the segment from another tier.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteTierFactory.java:
##########
@@ -75,7 +75,9 @@ public TierProducerAgent createProducerAgent(
@Override
public TierConsumerAgent createConsumerAgent(
List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
- TieredStorageNettyService nettyService) {
+ TieredStorageNettyService nettyService,
+ RemoteStorageScanner remoteStorageScanner,
+ PartitionFileReader partitionFileReader) {
// TODO, implement the remote tier consumer agent
Review Comment:
Shouldn't we be able to create the agent now?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageIdMappingUtils;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageConsumerSpec;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSegmentFinishPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSubpartitionPath;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /**
+ * Tuple3 containing partition id and subpartition id and segment id. The tuple3 is stored in
+ * queue and indicates the required segment file.
+ */
+ private final Queue<Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>>
+ requiredSegmentIds;
+
+ /**
+ * The channel indexes stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is related channel index.
+ */
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ channelIndexes;
+
+ private final String baseRemoteStoragePath;
+
+ private FileSystem remoteFileSystem;
+
+ private RemoteStorageScannerAvailabilityAndPriorityHelper helper;
+
+ public RemoteStorageScanner(
+ List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
+ String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.channelIndexes = new HashMap<>();
+ for (int index = 0; index < tieredStorageConsumerSpecs.size(); index++) {
+ TieredStorageConsumerSpec spec = tieredStorageConsumerSpecs.get(index);
+ channelIndexes
+ .computeIfAbsent(spec.getPartitionId(), ignore -> new HashMap<>())
+ .put(spec.getSubpartitionId(), index);
+ }
+ this.requiredSegmentIds = new LinkedBlockingDeque<>();
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.scheduleAtFixedRate(this, 0, 10, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Register a segment id to the {@link RemoteStorageScanner}. If the scanner discovers the
+ * segment file exists, it will trigger the reading process of the segment file.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void registerSegmentId(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ requiredSegmentIds.add(Tuple3.of(partitionId, subpartitionId, segmentId));
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the registered segment ids and check related file status. */
+ @Override
+ public void run() {
+ int segmentFileNumber = requiredSegmentIds.size();
+ while (segmentFileNumber-- > 0) {
+ Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>
+ partitionIdAndSubpartitionIdAndSegmentId =
+ checkNotNull(requiredSegmentIds.poll());
+ if (checkFileExist(
Review Comment:
Instead of checking the existence of individual segment files separately, I wonder if it makes sense to list all files at once and check for each segment.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/file/HashPartitionFileReader.java:
##########
@@ -40,8 +83,43 @@ public Buffer readBuffer(
MemorySegment memorySegment,
BufferRecycler recycler)
throws IOException {
- // TODO, implement the HashPartitionFileReader
- return null;
+ Tuple2<ReadableByteChannel, Integer> fileChannelAndSegmentId =
+ openedChannelAndSegmentIds
+ .computeIfAbsent(partitionId, ignore -> new HashMap<>())
+ .getOrDefault(subpartitionId, Tuple2.of(null, -1));
+ ReadableByteChannel channel = fileChannelAndSegmentId.f0;
+ if (channel == null || fileChannelAndSegmentId.f1 != segmentId) {
+ if (channel != null) {
+ channel.close();
+ }
+ channel = openNewChannel(partitionId, subpartitionId, segmentId);
+ if (channel == null) {
+ return null;
+ }
+ openedChannelAndSegmentIds
+ .get(partitionId)
Review Comment:
This `get` is unnecessary if we save `openedChannelAndSegmentIds.computeIfAbsent(...)` in a variable.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageIdMappingUtils;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageConsumerSpec;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava30.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSegmentFinishPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageUtils.generateSubpartitionPath;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /**
+ * Tuple3 containing partition id and subpartition id and segment id. The tuple3 is stored in
+ * queue and indicates the required segment file.
+ */
+ private final Queue<Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>>
+ requiredSegmentIds;
+
+ /**
+ * The channel indexes stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is related channel index.
+ */
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ channelIndexes;
+
+ private final String baseRemoteStoragePath;
+
+ private FileSystem remoteFileSystem;
+
+ private RemoteStorageScannerAvailabilityAndPriorityHelper helper;
+
+ public RemoteStorageScanner(
+ List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
+ String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.channelIndexes = new HashMap<>();
+ for (int index = 0; index < tieredStorageConsumerSpecs.size(); index++) {
+ TieredStorageConsumerSpec spec = tieredStorageConsumerSpecs.get(index);
+ channelIndexes
+ .computeIfAbsent(spec.getPartitionId(), ignore -> new HashMap<>())
+ .put(spec.getSubpartitionId(), index);
+ }
+ this.requiredSegmentIds = new LinkedBlockingDeque<>();
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.scheduleAtFixedRate(this, 0, 10, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Register a segment id to the {@link RemoteStorageScanner}. If the scanner discovers the
+ * segment file exists, it will trigger the reading process of the segment file.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void registerSegmentId(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ requiredSegmentIds.add(Tuple3.of(partitionId, subpartitionId, segmentId));
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the registered segment ids and check related file status. */
+ @Override
+ public void run() {
+ int segmentFileNumber = requiredSegmentIds.size();
+ while (segmentFileNumber-- > 0) {
+ Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>
+ partitionIdAndSubpartitionIdAndSegmentId =
+ checkNotNull(requiredSegmentIds.poll());
+ if (checkFileExist(
+ partitionIdAndSubpartitionIdAndSegmentId.f0,
+ partitionIdAndSubpartitionIdAndSegmentId.f1,
+ partitionIdAndSubpartitionIdAndSegmentId.f2)) {
+ helper.notifyAvailableAndPriority(
+ channelIndexes
+ .get(partitionIdAndSubpartitionIdAndSegmentId.f0)
+ .get(partitionIdAndSubpartitionIdAndSegmentId.f1),
+ false);
+ } else {
+ requiredSegmentIds.add(partitionIdAndSubpartitionIdAndSegmentId);
+ }
+ }
+ }
+
+ public void setupRemoteStorageScannerAvailabilityAndPriorityHelper(
+ RemoteStorageScannerAvailabilityAndPriorityHelper helper) {
+ this.helper = helper;
+ }
+
+ public void triggerNextRoundReading(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ boolean isPriority) {
+ helper.notifyAvailableAndPriority(
+ channelIndexes.get(partitionId).get(subpartitionId), isPriority);
+ }
Review Comment:
It's unclear to me what this method does. Why would we triggering reading through a scanner?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] WencongLiu closed pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "WencongLiu (via GitHub)" <gi...@apache.org>.
WencongLiu closed pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
URL: https://github.com/apache/flink/pull/22855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] WencongLiu commented on pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "WencongLiu (via GitHub)" <gi...@apache.org>.
WencongLiu commented on PR #22855:
URL: https://github.com/apache/flink/pull/22855#issuecomment-1633420229
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] reswqa commented on a diff in pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on code in PR #22855:
URL: https://github.com/apache/flink/pull/22855#discussion_r1265327084
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteTierConsumerAgent.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.PartitionFileReader;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityNotifier;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.TierConsumerAgent;
+import org.apache.flink.util.ExceptionUtils;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/** The data client is used to fetch data from remote tier. */
+public class RemoteTierConsumerAgent implements TierConsumerAgent {
+
+ private final RemoteStorageScanner remoteStorageScanner;
+
+ private final PartitionFileReader partitionFileReader;
+
+ /**
+ * The current reading buffer indexes and segment ids stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is buffer index and segment id.
+ */
+ private final Map<
+ TieredStoragePartitionId,
+ Map<TieredStorageSubpartitionId, Tuple2<Integer, Integer>>>
+ currentBufferIndexAndSegmentIds;
+
+ private final int bufferSizeBytes;
+
+ public RemoteTierConsumerAgent(
+ RemoteStorageScanner remoteStorageScanner,
+ PartitionFileReader partitionFileReader,
+ int bufferSizeBytes) {
+ this.remoteStorageScanner = remoteStorageScanner;
+ this.currentBufferIndexAndSegmentIds = new HashMap<>();
+ this.partitionFileReader = partitionFileReader;
+ this.bufferSizeBytes = bufferSizeBytes;
+ }
+
+ @Override
+ public void start() {
+ remoteStorageScanner.start();
+ }
+
+ @Override
+ public Optional<Buffer> getNextBuffer(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ // Get current segment id and buffer index.
+ Tuple2<Integer, Integer> bufferIndexAndSegmentId =
+ currentBufferIndexAndSegmentIds
+ .computeIfAbsent(partitionId, ignore -> new HashMap<>())
+ .getOrDefault(subpartitionId, Tuple2.of(0, -1));
+ int currentBufferIndex = bufferIndexAndSegmentId.f0;
+ int currentSegmentId = bufferIndexAndSegmentId.f1;
+ if (segmentId != currentSegmentId) {
+ remoteStorageScanner.watchSegment(partitionId, subpartitionId, segmentId);
+ }
+
+ // Read buffer from the partition file in remote storage.
+ MemorySegment memorySegment = MemorySegmentFactory.allocateUnpooledSegment(bufferSizeBytes);
+ Buffer buffer = null;
+ try {
+ buffer =
+ partitionFileReader.readBuffer(
+ partitionId,
+ subpartitionId,
+ segmentId,
+ currentBufferIndex,
+ memorySegment,
+ FreeingBufferRecycler.INSTANCE);
+ } catch (IOException e) {
+ memorySegment.free();
+ ExceptionUtils.rethrow(e, "Failed to read buffer from partition file.");
+ }
+ if (buffer != null) {
+ currentBufferIndexAndSegmentIds
+ .get(partitionId)
+ .put(subpartitionId, Tuple2.of(++currentBufferIndex, segmentId));
+ return Optional.of(buffer);
+ }
+ return Optional.empty();
Review Comment:
we may also need free segment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a diff in pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #22855:
URL: https://github.com/apache/flink/pull/22855#discussion_r1252639172
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/file/SegmentPartitionFileReader.java:
##########
@@ -42,8 +82,38 @@ public Buffer readBuffer(
MemorySegment memorySegment,
BufferRecycler recycler)
throws IOException {
- // TODO, implement the HashPartitionFileReader
- return null;
+ Map<TieredStorageSubpartitionId, Tuple2<ReadableByteChannel, Integer>> subpartitionInfo =
+ openedChannelAndSegmentIds.computeIfAbsent(partitionId, ignore -> new HashMap<>());
+ Tuple2<ReadableByteChannel, Integer> fileChannelAndSegmentId =
+ subpartitionInfo.getOrDefault(subpartitionId, Tuple2.of(null, -1));
+ ReadableByteChannel channel = fileChannelAndSegmentId.f0;
+ if (channel == null || fileChannelAndSegmentId.f1 != segmentId) {
+ if (channel != null) {
+ channel.close();
+ }
+ channel = openNewChannel(partitionId, subpartitionId, segmentId);
+ if (channel == null) {
+ return null;
Review Comment:
There's no indication that `PartitionFileReader#readBuffer` may return a `null`, neither from JavaDoc nor the `@Nullable` annotation.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/file/SegmentPartitionFileReader.java:
##########
@@ -42,8 +82,38 @@ public Buffer readBuffer(
MemorySegment memorySegment,
BufferRecycler recycler)
throws IOException {
- // TODO, implement the HashPartitionFileReader
- return null;
+ Map<TieredStorageSubpartitionId, Tuple2<ReadableByteChannel, Integer>> subpartitionInfo =
+ openedChannelAndSegmentIds.computeIfAbsent(partitionId, ignore -> new HashMap<>());
+ Tuple2<ReadableByteChannel, Integer> fileChannelAndSegmentId =
+ subpartitionInfo.getOrDefault(subpartitionId, Tuple2.of(null, -1));
+ ReadableByteChannel channel = fileChannelAndSegmentId.f0;
+ if (channel == null || fileChannelAndSegmentId.f1 != segmentId) {
+ if (channel != null) {
+ channel.close();
+ }
+ channel = openNewChannel(partitionId, subpartitionId, segmentId);
+ if (channel == null) {
+ return null;
+ }
+ subpartitionInfo.put(subpartitionId, Tuple2.of(channel, segmentId));
+ }
+
+ reusedHeaderBuffer.clear();
+ int bufferHeaderResult = channel.read(reusedHeaderBuffer);
+ if (bufferHeaderResult == -1) {
+ channel.close();
+ openedChannelAndSegmentIds.get(partitionId).remove(subpartitionId);
+ return new NetworkBuffer(memorySegment, recycler, Buffer.DataType.END_OF_SEGMENT);
+ }
+ reusedHeaderBuffer.rewind();
+ BufferHeader header = parseBufferHeader(reusedHeaderBuffer);
+ int dataBufferResult = channel.read(memorySegment.wrap(0, header.getLength()));
+ if (dataBufferResult == -1) {
+ throw new IOException("Empty data buffer is read.");
+ }
+ Buffer.DataType dataType = header.getDataType();
+ return new NetworkBuffer(
+ memorySegment, recycler, dataType, header.isCompressed(), header.getLength());
Review Comment:
This is a long piece of codes (over 30 lines). It would be nice to group it into some major steps with comments.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteTierFactory.java:
##########
@@ -89,8 +90,12 @@ public TierProducerAgent createProducerAgent(
@Override
public TierConsumerAgent createConsumerAgent(
List<TieredStorageConsumerSpec> tieredStorageConsumerSpecs,
- TieredStorageNettyService nettyService) {
- // TODO, implement the remote tier consumer agent
- return null;
+ TieredStorageNettyService nettyService,
+ RemoteStorageScanner remoteStorageScanner) {
Review Comment:
I don't think `RemoteStorageScanner` should be an argument for this common interface.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/TieredStorageConsumerClient.java:
##########
@@ -74,46 +76,54 @@ public void start() {
public Optional<Buffer> getNextBuffer(
TieredStoragePartitionId partitionId, TieredStorageSubpartitionId subpartitionId) {
- Tuple2<TierConsumerAgent, Integer> currentConsumerAgentAndSegmentId =
- currentConsumerAgentAndSegmentIds
- .computeIfAbsent(partitionId, ignore -> new HashMap<>())
- .getOrDefault(subpartitionId, Tuple2.of(null, 0));
+ Map<TieredStorageSubpartitionId, Tuple3<TierConsumerAgent, Integer, Integer>>
+ subpartitionInfo =
+ currentConsumerInfo.computeIfAbsent(partitionId, ignore -> new HashMap<>());
+ Tuple3<TierConsumerAgent, Integer, Integer> agentInfo =
+ subpartitionInfo.getOrDefault(subpartitionId, Tuple3.of(null, 0, 0));
Optional<Buffer> buffer = Optional.empty();
- if (currentConsumerAgentAndSegmentId.f0 == null) {
+ if (agentInfo.f0 == null) {
for (TierConsumerAgent tierConsumerAgent : tierConsumerAgents) {
- buffer =
- tierConsumerAgent.getNextBuffer(
- partitionId, subpartitionId, currentConsumerAgentAndSegmentId.f1);
+ buffer = tierConsumerAgent.getNextBuffer(partitionId, subpartitionId, agentInfo.f1);
if (buffer.isPresent()) {
- currentConsumerAgentAndSegmentIds
- .get(partitionId)
- .put(
- subpartitionId,
- Tuple2.of(
- tierConsumerAgent,
- currentConsumerAgentAndSegmentId.f1));
+ agentInfo.setField(tierConsumerAgent, 0);
break;
}
}
} else {
- buffer =
- currentConsumerAgentAndSegmentId.f0.getNextBuffer(
- partitionId, subpartitionId, currentConsumerAgentAndSegmentId.f1);
+ buffer = agentInfo.f0.getNextBuffer(partitionId, subpartitionId, agentInfo.f1);
}
if (!buffer.isPresent()) {
return Optional.empty();
}
Buffer bufferData = buffer.get();
+ if (retriever != null) {
+ retriever.retrieveAvailableAndPriority(
+ partitionId,
+ subpartitionId,
+ bufferData.getDataType().hasPriority(),
+ agentInfo.f2);
+ }
+ agentInfo.setField(agentInfo.f2 + 1, 2);
+ subpartitionInfo.put(subpartitionId, agentInfo);
if (bufferData.getDataType() == Buffer.DataType.END_OF_SEGMENT) {
- currentConsumerAgentAndSegmentIds
- .get(partitionId)
- .put(subpartitionId, Tuple2.of(null, currentConsumerAgentAndSegmentId.f1 + 1));
+ agentInfo.setField(null, 0);
+ agentInfo.setField(agentInfo.f1 + 1, 1);
+ subpartitionInfo.put(subpartitionId, agentInfo);
bufferData.recycleBuffer();
return getNextBuffer(partitionId, subpartitionId);
}
return Optional.of(bufferData);
}
+ public void registerAvailabilityAndPriorityRetriever(
+ AvailabilityAndPriorityRetriever retriever) {
+ this.retriever = retriever;
+ if (remoteStorageScanner != null) {
+ remoteStorageScanner.registerAvailabilityAndPriorityRetriever(retriever);
+ }
+ }
Review Comment:
1. The name is confusing. I'd suggest to use `listener` rather than `retriever`.
2. The argument type is an inner class of `SingleInputGate`.
3. I think this entire change (i.e., introducing the availability and priority mechanism as part of the tiered storage client protocol, and migrate the netty related implementations) should be a separate commit in prior to implementing the remote tier consumer agent.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.AvailabilityAndPriorityRetriever;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 500ms. */
+ private static final int INITIAL_SCAN_INTERVAL = 500;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /**
+ * Tuple3 containing partition id and subpartition id and segment id. The tuple3 is stored in
+ * queue and indicates the required segment file.
+ */
+ private final Queue<Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private FileSystem remoteFileSystem;
+
+ private AvailabilityAndPriorityRetriever retriever;
+
+ private int attemptNumber = 0;
+
+ public RemoteStorageScanner(String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.requiredSegmentIds = new LinkedBlockingDeque<>();
+ this.scannedMaxSegmentIds = new HashMap<>();
+ this.scanStrategy = new ScanStrategy(INITIAL_SCAN_INTERVAL, MAX_SCAN_INTERVAL);
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.schedule(
+ this, scanStrategy.getInterval(attemptNumber), TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Register a segment id to the {@link RemoteStorageScanner}. If the scanner discovers the
+ * segment file exists, it will trigger the reading process of the segment file.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void registerSegmentId(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ requiredSegmentIds.add(Tuple3.of(partitionId, subpartitionId, segmentId));
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the registered segment ids and check related file status. */
+ @Override
+ public void run() {
+ int segmentFileNumber = requiredSegmentIds.size();
+ boolean scanned = false;
+ while (segmentFileNumber-- > 0) {
+ Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer> ids =
+ checkNotNull(requiredSegmentIds.poll());
+ TieredStoragePartitionId partitionId = ids.f0;
+ TieredStorageSubpartitionId subpartitionId = ids.f1;
+ int requiredSegmentId = ids.f2;
+ int maxSegmentId =
+ scannedMaxSegmentIds
+ .computeIfAbsent(partitionId, ignore -> new HashMap<>())
+ .getOrDefault(subpartitionId, -1);
+ if (maxSegmentId >= requiredSegmentId) {
+ scanned = true;
+ retriever.retrieveAvailableAndPriority(partitionId, subpartitionId, false, null);
+ } else {
+ scanMaxSegmentId(partitionId, subpartitionId);
+ requiredSegmentIds.add(ids);
Review Comment:
Again, what if the segment is no longer needed?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.AvailabilityAndPriorityRetriever;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 500ms. */
+ private static final int INITIAL_SCAN_INTERVAL = 500;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL = 10_000;
Review Comment:
A good practice is to always add the unit as part of the name for unit-sensitive variables such as time and memory.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.AvailabilityAndPriorityRetriever;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to register the required segment id and
+ * monitor the existence status of related segment files. If the segment file is found, it will
+ * trigger the reading process of the file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 500ms. */
+ private static final int INITIAL_SCAN_INTERVAL = 500;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /**
+ * Tuple3 containing partition id and subpartition id and segment id. The tuple3 is stored in
+ * queue and indicates the required segment file.
+ */
+ private final Queue<Tuple3<TieredStoragePartitionId, TieredStorageSubpartitionId, Integer>>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private FileSystem remoteFileSystem;
+
+ private AvailabilityAndPriorityRetriever retriever;
+
+ private int attemptNumber = 0;
+
+ public RemoteStorageScanner(String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.requiredSegmentIds = new LinkedBlockingDeque<>();
+ this.scannedMaxSegmentIds = new HashMap<>();
+ this.scanStrategy = new ScanStrategy(INITIAL_SCAN_INTERVAL, MAX_SCAN_INTERVAL);
+ try {
+ this.remoteFileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.schedule(
+ this, scanStrategy.getInterval(attemptNumber), TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Register a segment id to the {@link RemoteStorageScanner}. If the scanner discovers the
+ * segment file exists, it will trigger the reading process of the segment file.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void registerSegmentId(
Review Comment:
I'd suggest the name `watchSegment`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] reswqa commented on a diff in pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on code in PR #22855:
URL: https://github.com/apache/flink/pull/22855#discussion_r1263431982
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/file/SegmentPartitionFileReader.java:
##########
@@ -42,8 +82,44 @@ public Buffer readBuffer(
MemorySegment memorySegment,
BufferRecycler recycler)
throws IOException {
- // TODO, implement the SegmentPartitionFileReader
- return null;
+
+ // Get the channel of the segment file for a subpartition.
+ Map<TieredStorageSubpartitionId, Tuple2<ReadableByteChannel, Integer>> subpartitionInfo =
+ openedChannelAndSegmentIds.computeIfAbsent(partitionId, ignore -> new HashMap<>());
+ Tuple2<ReadableByteChannel, Integer> fileChannelAndSegmentId =
+ subpartitionInfo.getOrDefault(subpartitionId, Tuple2.of(null, -1));
+ ReadableByteChannel channel = fileChannelAndSegmentId.f0;
+
+ // Create the channel if there is a new segment file for a subpartition.
+ if (channel == null || fileChannelAndSegmentId.f1 != segmentId) {
+ if (channel != null) {
+ channel.close();
+ }
+ channel = openNewChannel(partitionId, subpartitionId, segmentId);
+ if (channel == null) {
+ // return null if the segment file doesn't exist.
+ return null;
+ }
+ subpartitionInfo.put(subpartitionId, Tuple2.of(channel, segmentId));
+ }
+
+ // Try to read a buffer from the channel.
+ reusedHeaderBuffer.clear();
+ int bufferHeaderResult = channel.read(reusedHeaderBuffer);
+ if (bufferHeaderResult == -1) {
+ channel.close();
+ openedChannelAndSegmentIds.get(partitionId).remove(subpartitionId);
+ return new NetworkBuffer(memorySegment, recycler, Buffer.DataType.END_OF_SEGMENT);
+ }
+ reusedHeaderBuffer.rewind();
Review Comment:
If `reusedHeaderBuffer.flip()` also applies, I would recommend it more.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/file/SegmentPartitionFileReader.java:
##########
@@ -42,8 +82,44 @@ public Buffer readBuffer(
MemorySegment memorySegment,
BufferRecycler recycler)
throws IOException {
- // TODO, implement the SegmentPartitionFileReader
- return null;
+
+ // Get the channel of the segment file for a subpartition.
+ Map<TieredStorageSubpartitionId, Tuple2<ReadableByteChannel, Integer>> subpartitionInfo =
+ openedChannelAndSegmentIds.computeIfAbsent(partitionId, ignore -> new HashMap<>());
+ Tuple2<ReadableByteChannel, Integer> fileChannelAndSegmentId =
+ subpartitionInfo.getOrDefault(subpartitionId, Tuple2.of(null, -1));
+ ReadableByteChannel channel = fileChannelAndSegmentId.f0;
+
+ // Create the channel if there is a new segment file for a subpartition.
+ if (channel == null || fileChannelAndSegmentId.f1 != segmentId) {
+ if (channel != null) {
+ channel.close();
+ }
+ channel = openNewChannel(partitionId, subpartitionId, segmentId);
+ if (channel == null) {
+ // return null if the segment file doesn't exist.
+ return null;
+ }
+ subpartitionInfo.put(subpartitionId, Tuple2.of(channel, segmentId));
+ }
+
+ // Try to read a buffer from the channel.
+ reusedHeaderBuffer.clear();
+ int bufferHeaderResult = channel.read(reusedHeaderBuffer);
+ if (bufferHeaderResult == -1) {
+ channel.close();
+ openedChannelAndSegmentIds.get(partitionId).remove(subpartitionId);
+ return new NetworkBuffer(memorySegment, recycler, Buffer.DataType.END_OF_SEGMENT);
+ }
+ reusedHeaderBuffer.rewind();
+ BufferHeader header = parseBufferHeader(reusedHeaderBuffer);
+ int dataBufferResult = channel.read(memorySegment.wrap(0, header.getLength()));
+ if (dataBufferResult == -1) {
+ throw new IOException("Empty data buffer is read.");
Review Comment:
Should we release(close) this channel before throw exception?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteTierConsumerAgent.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.PartitionFileReader;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityNotifier;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.TierConsumerAgent;
+import org.apache.flink.util.ExceptionUtils;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/** The data client is used to fetch data from remote tier. */
+public class RemoteTierConsumerAgent implements TierConsumerAgent {
+
+ private final RemoteStorageScanner remoteStorageScanner;
+
+ private final PartitionFileReader partitionFileReader;
+
+ /**
+ * The current reading buffer indexes and segment ids stored in map.
+ *
+ * <p>The key is partition id and subpartition id. The value is buffer index and segment id.
+ */
+ private final Map<
+ TieredStoragePartitionId,
+ Map<TieredStorageSubpartitionId, Tuple2<Integer, Integer>>>
+ currentBufferIndexAndSegmentIds;
+
+ private final int bufferSizeBytes;
+
+ public RemoteTierConsumerAgent(
+ RemoteStorageScanner remoteStorageScanner,
+ PartitionFileReader partitionFileReader,
+ int bufferSizeBytes) {
+ this.remoteStorageScanner = remoteStorageScanner;
+ this.currentBufferIndexAndSegmentIds = new HashMap<>();
+ this.partitionFileReader = partitionFileReader;
+ this.bufferSizeBytes = bufferSizeBytes;
+ }
+
+ @Override
+ public void start() {
+ remoteStorageScanner.start();
+ }
+
+ @Override
+ public Optional<Buffer> getNextBuffer(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ // Get current segment id and buffer index.
+ Tuple2<Integer, Integer> bufferIndexAndSegmentId =
+ currentBufferIndexAndSegmentIds
+ .computeIfAbsent(partitionId, ignore -> new HashMap<>())
+ .getOrDefault(subpartitionId, Tuple2.of(0, -1));
+ int currentBufferIndex = bufferIndexAndSegmentId.f0;
+ int currentSegmentId = bufferIndexAndSegmentId.f1;
+ if (segmentId != currentSegmentId) {
+ remoteStorageScanner.watchSegment(partitionId, subpartitionId, segmentId);
+ }
+
+ // Read buffer from the partition file in remote storage.
+ MemorySegment memorySegment = MemorySegmentFactory.allocateUnpooledSegment(bufferSizeBytes);
Review Comment:
We should also free this segment if something unexpected happens. I know that the current implementation will not cause problems as this `byte[]` will be released during gc even though we manually free it. But this is not a good pattern.
##########
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScannerTest.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionID;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageIdMappingUtils;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.RejectedExecutionException;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.writeSegmentFinishFile;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+/** Tests for {@link RemoteStorageScanner}. */
+class RemoteStorageScannerTest {
+
+ private static final TieredStoragePartitionId DEFAULT_PARTITION_ID =
+ TieredStorageIdMappingUtils.convertId(new ResultPartitionID());
+
+ private static final TieredStorageSubpartitionId DEFAULT_SUBPARTITION_ID =
+ new TieredStorageSubpartitionId(0);
+
+ @TempDir private java.nio.file.Path tempFolder;
Review Comment:
```suggestion
@TempDir private File tempFolder;
```
Since we will definitely convert it into a `File`, why not use it directly? Other tests are the same.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityNotifier;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to watch the required segments and scan
+ * the existence status of the segments. If the segment file is found, it will notify the
+ * availability of segment file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 100ms. */
+ private static final int INITIAL_SCAN_INTERVAL_MS = 100;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL_MS = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /** The key is partition id and subpartition id, the value is required segment id. */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private final FileSystem remoteFileSystem;
+
+ private AvailabilityNotifier notifier;
Review Comment:
This should be marked as `Nullable`.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/file/SegmentPartitionFileReader.java:
##########
@@ -42,8 +82,44 @@ public Buffer readBuffer(
MemorySegment memorySegment,
BufferRecycler recycler)
throws IOException {
- // TODO, implement the SegmentPartitionFileReader
- return null;
+
+ // Get the channel of the segment file for a subpartition.
+ Map<TieredStorageSubpartitionId, Tuple2<ReadableByteChannel, Integer>> subpartitionInfo =
+ openedChannelAndSegmentIds.computeIfAbsent(partitionId, ignore -> new HashMap<>());
+ Tuple2<ReadableByteChannel, Integer> fileChannelAndSegmentId =
+ subpartitionInfo.getOrDefault(subpartitionId, Tuple2.of(null, -1));
+ ReadableByteChannel channel = fileChannelAndSegmentId.f0;
+
+ // Create the channel if there is a new segment file for a subpartition.
+ if (channel == null || fileChannelAndSegmentId.f1 != segmentId) {
+ if (channel != null) {
+ channel.close();
+ }
+ channel = openNewChannel(partitionId, subpartitionId, segmentId);
+ if (channel == null) {
+ // return null if the segment file doesn't exist.
+ return null;
+ }
+ subpartitionInfo.put(subpartitionId, Tuple2.of(channel, segmentId));
+ }
+
+ // Try to read a buffer from the channel.
+ reusedHeaderBuffer.clear();
+ int bufferHeaderResult = channel.read(reusedHeaderBuffer);
+ if (bufferHeaderResult == -1) {
+ channel.close();
+ openedChannelAndSegmentIds.get(partitionId).remove(subpartitionId);
+ return new NetworkBuffer(memorySegment, recycler, Buffer.DataType.END_OF_SEGMENT);
+ }
+ reusedHeaderBuffer.rewind();
+ BufferHeader header = parseBufferHeader(reusedHeaderBuffer);
+ int dataBufferResult = channel.read(memorySegment.wrap(0, header.getLength()));
+ if (dataBufferResult == -1) {
Review Comment:
Maybe we should check the `dataBufferResult` is equals to `header.getLength()` instead of empty.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityNotifier;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to watch the required segments and scan
+ * the existence status of the segments. If the segment file is found, it will notify the
+ * availability of segment file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 100ms. */
+ private static final int INITIAL_SCAN_INTERVAL_MS = 100;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL_MS = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
Review Comment:
What are we going to do with UncaughtException?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22855: [FLINK-31642][network] Introduce the RemoteTierConsumerAgent
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22855:
URL: https://github.com/apache/flink/pull/22855#issuecomment-1605862481
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d85e07af70665179feaf8f68b2e8d5ff75811884",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d85e07af70665179feaf8f68b2e8d5ff75811884",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d85e07af70665179feaf8f68b2e8d5ff75811884 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a diff in pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #22855:
URL: https://github.com/apache/flink/pull/22855#discussion_r1259375162
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/SingleInputGate.java:
##########
@@ -1146,21 +1151,35 @@ private void setupTieredStorageNettyService(
int channelIndex = index;
channelSuppliers.add(() -> channels[channelIndex]);
}
- nettyService.setupInputChannels(
- tieredStorageConsumerSpecs,
- channelSuppliers,
- new NettyConnectionReaderAvailabilityAndPriorityHelper() {
- @Override
- public void notifyReaderAvailableAndPriority(
- int channelIndex, boolean isPriority) {
- queueChannel(channels[channelIndex], null, isPriority);
- }
+ nettyService.setupInputChannels(tieredStorageConsumerSpecs, channelSuppliers);
+ }
- @Override
- public void updatePrioritySequenceNumber(int channelIndex, int sequenceNumber) {
- lastPrioritySequenceNumber[channelIndex] = sequenceNumber;
- }
- });
+ /** The default implementation of {@link AvailabilityAndPriorityNotifier}. */
+ private class AvailabilityAndPriorityNotifierImpl implements AvailabilityAndPriorityNotifier {
+
+ private final Map<TieredStoragePartitionId, Map<TieredStorageSubpartitionId, Integer>>
+ channelIndexes;
+
+ public AvailabilityAndPriorityNotifierImpl() {
Review Comment:
Should be private.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/storage/AvailabilityAndPriorityNotifier.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage;
+
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+
+/**
+ * {@link AvailabilityAndPriorityNotifier} is used to notify the availability and priority status of
+ * a specific partition and subpartition in tiered storage.
+ */
+public interface AvailabilityAndPriorityNotifier {
+
+ /**
+ * Notify the availability and priority status of a specific partition and subpartition. This
+ * method will be invoked by {@link TieredStorageConsumerClient} if the subpartition in tiered
+ * storage has more available buffers or should be read with priority.
+ *
+ * @param partitionId the partition id.
+ * @param subpartitionId the subpartition id.
+ * @param isPriority the subpartition will be consumed with priority if the value is true
+ * otherwise not.
+ * @param prioritySequenceNumber the sequence number of priority buffer.
+ */
+ void notifyAvailableAndPriority(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ boolean isPriority,
+ Integer prioritySequenceNumber);
Review Comment:
1. I'd suggest the name `priorityBufferIndex`, to align the terminology.
2. Why `Integer`, rather than `int`?
3. What is this value when `isPriority` is `false`?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskTierConsumerAgent.java:
##########
@@ -34,14 +39,20 @@
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
/** The data client is used to fetch data from disk tier. */
public class DiskTierConsumerAgent implements TierConsumerAgent {
private final Map<
TieredStoragePartitionId,
- Map<TieredStorageSubpartitionId, CompletableFuture<NettyConnectionReader>>>
+ Map<
+ TieredStorageSubpartitionId,
+ Tuple2<CompletableFuture<NettyConnectionReader>, Integer>>>
nettyConnectionReaders = new HashMap<>();
+ @Nullable private AvailabilityAndPriorityNotifier notifier;
Review Comment:
What does the null-value mean?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/disk/DiskTierConsumerAgent.java:
##########
@@ -62,20 +74,36 @@ public void start() {
// noop
}
+ @Override
+ public void registerAvailabilityAndPriorityNotifier(AvailabilityAndPriorityNotifier notifier) {
+ this.notifier = notifier;
+ }
+
@Override
public Optional<Buffer> getNextBuffer(
TieredStoragePartitionId partitionId,
TieredStorageSubpartitionId subpartitionId,
int segmentId) {
+ Optional<Buffer> buffer = Optional.empty();
+ Tuple2<CompletableFuture<NettyConnectionReader>, Integer> readerAndBufferIndex =
+ nettyConnectionReaders.get(partitionId).get(subpartitionId);
try {
- return nettyConnectionReaders
- .get(partitionId)
- .get(subpartitionId)
- .get()
- .readBuffer(segmentId);
+ buffer = readerAndBufferIndex.f0.get().readBuffer(segmentId);
} catch (InterruptedException | ExecutionException e) {
- throw new RuntimeException("Failed to get next buffer.", e);
+ ExceptionUtils.rethrow(e, "Failed to read buffer from memory tier.");
}
+ buffer.ifPresent(
+ value -> {
+ boolean isPriority = value.getDataType().hasPriority();
+ checkNotNull(notifier)
+ .notifyAvailableAndPriority(
+ partitionId,
+ subpartitionId,
+ isPriority,
+ isPriority ? readerAndBufferIndex.f1 : null);
+ readerAndBufferIndex.f1 += 1;
Review Comment:
How the three tiers use the notifier seems quite similar. I wonder why this is inside each tier? Is it simply because the consumer client doesn't know the buffer index? If true, we may consider to include buffer index as part of the return value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] xintongsong commented on a diff in pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on code in PR #22855:
URL: https://github.com/apache/flink/pull/22855#discussion_r1260830009
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/TierConsumerAgent.java:
##########
@@ -47,6 +48,13 @@ Optional<Buffer> getNextBuffer(
TieredStorageSubpartitionId subpartitionId,
int segmentId);
+ /**
+ * Register the notifier to notify the availability of a subpartition.
+ *
+ * @param notifier to notify availability and priority.
+ */
+ void registerAvailabilityAndPriorityNotifier(AvailabilityNotifier notifier);
Review Comment:
Name needs update.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/hybrid/tiered/tier/remote/RemoteStorageScanner.java:
##########
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition.hybrid.tiered.tier.remote;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.fs.FileStatus;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStoragePartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.common.TieredStorageSubpartitionId;
+import org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.AvailabilityNotifier;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.apache.flink.shaded.guava31.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentFinishDirPath;
+import static org.apache.flink.runtime.io.network.partition.hybrid.tiered.file.SegmentPartitionFile.getSegmentPath;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * The {@link RemoteStorageScanner} is introduced to notify asynchronously for file reading on
+ * remote storage. Asynchronous notifications will prevent {@link RemoteTierConsumerAgent} from
+ * repeatedly attempting to read remote files and reduce CPU consumption.
+ *
+ * <p>It will be invoked by {@link RemoteTierConsumerAgent} to watch the required segments and scan
+ * the existence status of the segments. If the segment file is found, it will notify the
+ * availability of segment file.
+ */
+public class RemoteStorageScanner implements Runnable {
+
+ /** The initial scan interval is 100ms. */
+ private static final int INITIAL_SCAN_INTERVAL_MS = 100;
+
+ /** The max scan interval is 10000ms. */
+ private static final int MAX_SCAN_INTERVAL_MS = 10_000;
+
+ /** Executor to scan the existence status of segment files on remote storage. */
+ private final ScheduledExecutorService scannerExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setNameFormat("remote storage file scanner")
+ .build());
+
+ /** The key is partition id and subpartition id, the value is required segment id. */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ requiredSegmentIds;
+
+ /**
+ * The key is partition id and subpartition id, the value is max id of written segment files in
+ * the subpartition.
+ */
+ private final Map<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>
+ scannedMaxSegmentIds;
+
+ private final String baseRemoteStoragePath;
+
+ private final ScanStrategy scanStrategy;
+
+ private final FileSystem remoteFileSystem;
+
+ private AvailabilityNotifier notifier;
+
+ private int lastInterval = INITIAL_SCAN_INTERVAL_MS;
+
+ public RemoteStorageScanner(String baseRemoteStoragePath) {
+ this.baseRemoteStoragePath = baseRemoteStoragePath;
+ this.requiredSegmentIds = new ConcurrentHashMap<>();
+ this.scannedMaxSegmentIds = new ConcurrentHashMap<>();
+ this.scanStrategy = new ScanStrategy(MAX_SCAN_INTERVAL_MS);
+ this.remoteFileSystem = createFileSystem();
+ }
+
+ private FileSystem createFileSystem() {
+ FileSystem fileSystem = null;
+ try {
+ fileSystem = new Path(baseRemoteStoragePath).getFileSystem();
+ } catch (IOException e) {
+ ExceptionUtils.rethrow(
+ e, "Failed to initialize file system on the path: " + baseRemoteStoragePath);
+ }
+ return fileSystem;
+ }
+
+ /** Start the executor. */
+ public void start() {
+ scannerExecutor.schedule(this, lastInterval, TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Watch the segment for a specific subpartition in the {@link RemoteStorageScanner}.
+ *
+ * <p>If a segment with a larger or equal id already exists, the current segment won't be
+ * watched.
+ *
+ * <p>If a segment with a smaller segment id is still being watched, the current segment will
+ * replace it because the smaller segment should have been consumed. This method ensures that
+ * only one segment file can be watched for each subpartition.
+ *
+ * @param partitionId is the id of partition.
+ * @param subpartitionId is the id of subpartition.
+ * @param segmentId is the id of segment.
+ */
+ public void watchSegment(
+ TieredStoragePartitionId partitionId,
+ TieredStorageSubpartitionId subpartitionId,
+ int segmentId) {
+ Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId> key =
+ Tuple2.of(partitionId, subpartitionId);
+ scannedMaxSegmentIds.compute(
+ key,
+ (segmentKey, maxSegmentId) -> {
+ if (maxSegmentId == null || maxSegmentId < segmentId) {
+ requiredSegmentIds.put(segmentKey, segmentId);
+ }
+ return maxSegmentId;
+ });
+ }
+
+ /** Close the executor. */
+ public void close() {
+ scannerExecutor.shutdownNow();
+ }
+
+ /** Iterate the watched segment ids and check related file status. */
+ @Override
+ public void run() {
+ Iterator<Map.Entry<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer>>
+ iterator = requiredSegmentIds.entrySet().iterator();
+ boolean scanned = false;
+ while (iterator.hasNext()) {
+ Map.Entry<Tuple2<TieredStoragePartitionId, TieredStorageSubpartitionId>, Integer> ids =
+ iterator.next();
+ TieredStoragePartitionId partitionId = ids.getKey().f0;
+ TieredStorageSubpartitionId subpartitionId = ids.getKey().f1;
+ int requiredSegmentId = ids.getValue();
+ int maxSegmentId = scannedMaxSegmentIds.getOrDefault(ids.getKey(), -1);
+ if (maxSegmentId >= requiredSegmentId
+ && checkSegmentExist(partitionId, subpartitionId, requiredSegmentId)) {
+ scanned = true;
+ iterator.remove();
+ notifier.notifyAvailable(partitionId, subpartitionId);
+ } else {
+ // The segment should be watched again because it's not found.
+ // If the segment belongs to other tiers and has been consumed, the segment will be
+ // replaced by newly watched segment with larger segment id. This logic is ensured
+ // by the method {@code watchSegment}.
+ scanMaxSegmentId(partitionId, subpartitionId);
+ }
+ }
+ lastInterval = scanned ? INITIAL_SCAN_INTERVAL_MS : scanStrategy.getInterval(lastInterval);
+ start();
+ }
+
+ public void registerAvailabilityAndPriorityNotifier(AvailabilityNotifier retriever) {
Review Comment:
Same here for the name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] reswqa merged pull request #22855: [FLINK-31647][network] Introduce the RemoteTierConsumerAgent
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa merged PR #22855:
URL: https://github.com/apache/flink/pull/22855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org