You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/07/15 08:12:20 UTC
[GitHub] [flink] Thesharing opened a new pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Thesharing opened a new pull request #16498:
URL: https://github.com/apache/flink/pull/16498
This is a draft working in progress.
## What is the purpose of the change
*This pull request intends to add a limit size for the blob cache on TaskExecutor. If the size limits exceeds, blobs are deleted in LRU order.*
## Brief change log
- *Introduce BlobCacheSizeTracker to track the size of blob caches.*
## Verifying this change
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (100MB)*
- *Extended integration test for recovery after master (JobManager) failure*
- *Added test that validates that TaskInfo is transferred only once across recoveries*
- *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**yes** / no / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677165668
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobService.java
##########
@@ -45,4 +46,21 @@
* @throws IOException if any other error occurs when retrieving the file
*/
File getFile(JobID jobId, PermanentBlobKey key) throws IOException;
+
+ /**
+ * Directly read the file associated with the provided job ID and blob key.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read when
Review comment:
>>> {@code readFile} makes sure that ..
I think it's not very accurate because `readFile` does not guarantee that but it provides the possibility to achieve that, if the file reading and file retrieving happens in the same write lock.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
Review comment:
It's not needed because it is implemented the same by `DefaultExecutionGraphDeploymentWithBlobServerTest` already
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
+ if (blobServer != null) {
+ blobServer.close();
+ }
+ }
+
+ @Test
+ public void testDeployTasksWithMinimumSizeLimit() throws Exception {
Review comment:
Why do we need to add this test given that we already have `DefaultExecutionGraphDeploymentTest#testBuildDeploymentDescriptor`?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
+ if (blobServer != null) {
+ blobServer.close();
+ }
+ }
+
+ @Test
+ public void testDeployTasksWithMinimumSizeLimit() throws Exception {
+
+ final int numberOfVertices = 4;
+ final int parallelism = 10;
+
+ final ExecutionGraph eg = createAndSetupExecutionGraph(numberOfVertices, parallelism);
+
+ final SimpleAckingTaskManagerGateway taskManagerGateway =
+ new SimpleAckingTaskManagerGateway();
+ final BlockingQueue<TaskDeploymentDescriptor> tdds =
+ new ArrayBlockingQueue<>(numberOfVertices * parallelism);
+ taskManagerGateway.setSubmitConsumer(
+ FunctionUtils.uncheckedConsumer(
+ taskDeploymentDescriptor -> {
+ taskDeploymentDescriptor.loadBigData(blobCache);
+ tdds.offer(taskDeploymentDescriptor);
+ }));
+
+ for (ExecutionJobVertex ejv : eg.getVerticesTopologically()) {
+ for (ExecutionVertex ev : ejv.getTaskVertices()) {
+
+ assertEquals(ExecutionState.CREATED, ev.getExecutionState());
+
+ LogicalSlot slot =
+ new TestingLogicalSlotBuilder()
+ .setTaskManagerGateway(taskManagerGateway)
+ .createTestingLogicalSlot();
+ ev.getCurrentExecutionAttempt()
+ .registerProducedPartitions(slot.getTaskManagerLocation(), true)
+ .get();
+ ev.deployToSlot(slot);
+ assertEquals(ExecutionState.DEPLOYING, ev.getExecutionState());
+
+ TaskDeploymentDescriptor tdd = tdds.take();
+ assertNotNull(tdd);
+
+ List<InputGateDeploymentDescriptor> igdds = tdd.getInputGates();
+ assertEquals(ev.getAllConsumedPartitionGroups().size(), igdds.size());
+
+ if (igdds.size() > 0) {
+ assertShuffleDescriptorsEqual(igdds.get(0), ev.getConsumedPartitionGroup(0));
+ }
+ }
+ }
+ }
+
+ private ExecutionGraph createAndSetupExecutionGraph(int numberOfVertices, int parallelism)
+ throws JobException, JobExecutionException {
+
+ final List<JobVertex> vertices = new ArrayList<>();
+
+ for (int i = 0; i < numberOfVertices; i++) {
+ JobVertex vertex = new JobVertex(String.format("v%d", i + 1), new JobVertexID());
+ vertex.setParallelism(parallelism);
+ vertex.setInvokableClass(BatchTask.class);
+ vertices.add(vertex);
+ }
+
+ for (int i = 1; i < numberOfVertices; i++) {
+ vertices.get(i)
+ .connectNewDataSetAsInput(
+ vertices.get(i - 1),
+ DistributionPattern.ALL_TO_ALL,
+ ResultPartitionType.BLOCKING);
+ }
+
+ final JobGraph jobGraph =
+ JobGraphTestUtils.batchJobGraph(vertices.toArray(new JobVertex[0]));
+
+ final DirectScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final DefaultExecutionGraph eg =
+ TestingDefaultExecutionGraphBuilder.newBuilder()
+ .setJobGraph(jobGraph)
+ .setFutureExecutor(executor)
+ .setIoExecutor(executor)
+ .setBlobWriter(blobWriter)
+ .build();
+
+ eg.start(ComponentMainThreadExecutorServiceAdapter.forMainThread());
+
+ return eg;
+ }
+
+ private static void assertShuffleDescriptorsEqual(
Review comment:
Looks to me a name `checkShuffleDescriptors` is more accurate.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
Review comment:
`DefaultExecutionGraphDeploymentWithSizeLimitTest` -> `DefaultExecutionGraphDeploymentWithBlobSizeLimitTest `
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
Review comment:
>>> To avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first and then hold the lock here.
I cannot come up with the case that how deadlocks can happen in this case. However, I can see that inconsistency can happen if track/untrack/checkLimit are not guarded by write/read lock of blob cache.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -102,6 +123,9 @@ public PermanentBlobCache(
this.cleanupInterval = blobClientConfig.getLong(BlobServerOptions.CLEANUP_INTERVAL) * 1000;
this.cleanupTimer.schedule(
new PermanentBlobCleanupTask(), cleanupInterval, cleanupInterval);
+
+ // Configure the size limit tracker
Review comment:
This comment does not add extra info and thus is not needed I think
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
+ BlobKey blobKey,
+ File localFile,
+ Logger log,
+ @Nullable BlobStore blobStore)
+ throws IOException {
+
+ // Check the size limit and delete the files that exceeds the limit
+ final long sizeOfIncomingFile = incomingFile.length();
+ final List<Tuple2<JobID, BlobKey>> blobsToDelete =
+ blobCacheSizeTracker.checkLimit(sizeOfIncomingFile);
+
+ for (Tuple2<JobID, BlobKey> key : blobsToDelete) {
+ deleteFile(key.f0, key.f1);
+ blobCacheSizeTracker.untrack(key);
Review comment:
We should not `untrack` the blob if the deletion failed.
But I prefer to not block the new blob adding if this happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r673859802
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrency modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link AbstractBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ private final long capacity;
Review comment:
> name it as `sizeLimit` would be easier for understanding
Thank you for pointing this out. Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-884085778
Hi @zhuzhurk. I've refactor the code based on the lastest idea we've discussed before. Would you mind re-reviewing it if you got any free time? Thank you in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r671089379
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/BlobServerOptions.java
##########
@@ -118,4 +118,10 @@
key("blob.client.connect.timeout")
.defaultValue(0)
.withDescription("The connection timeout in milliseconds for the blob client.");
+
+ /** The maximum size limit of blob cache on TaskExecutor. */
+ public static final ConfigOption<Long> BLOB_CACHE_SIZE_LIMIT =
+ key("blob.cache.size.limit")
+ .defaultValue(1_024 * 1_024 * 1_024 * 10L)
+ .withDescription("The maximum size limit of blob cache for each TaskExecutor.");
Review comment:
the description needs clearly state how it affects permanent blob and transient blob size.
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/BlobServerOptions.java
##########
@@ -118,4 +118,10 @@
key("blob.client.connect.timeout")
.defaultValue(0)
.withDescription("The connection timeout in milliseconds for the blob client.");
+
+ /** The maximum size limit of blob cache on TaskExecutor. */
+ public static final ConfigOption<Long> BLOB_CACHE_SIZE_LIMIT =
+ key("blob.cache.size.limit")
+ .defaultValue(1_024 * 1_024 * 1_024 * 10L)
+ .withDescription("The maximum size limit of blob cache for each TaskExecutor.");
Review comment:
the description needs to clearly state how it affects permanent blob and transient blob size limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124",
"triggerID" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21133",
"triggerID" : "888721980",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 01f718e46e76efe7f74fa927a82a5329f5ef012b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21133)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 7ce8a08f3d996eee8c4ac7ac2de4618152099ac7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038)
* b9d70021d437374b456556dce817d203bd4f21e2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677282028
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
+ if (blobServer != null) {
+ blobServer.close();
+ }
+ }
+
+ @Test
+ public void testDeployTasksWithMinimumSizeLimit() throws Exception {
+
+ final int numberOfVertices = 4;
+ final int parallelism = 10;
+
+ final ExecutionGraph eg = createAndSetupExecutionGraph(numberOfVertices, parallelism);
+
+ final SimpleAckingTaskManagerGateway taskManagerGateway =
+ new SimpleAckingTaskManagerGateway();
+ final BlockingQueue<TaskDeploymentDescriptor> tdds =
+ new ArrayBlockingQueue<>(numberOfVertices * parallelism);
+ taskManagerGateway.setSubmitConsumer(
+ FunctionUtils.uncheckedConsumer(
+ taskDeploymentDescriptor -> {
+ taskDeploymentDescriptor.loadBigData(blobCache);
+ tdds.offer(taskDeploymentDescriptor);
+ }));
+
+ for (ExecutionJobVertex ejv : eg.getVerticesTopologically()) {
+ for (ExecutionVertex ev : ejv.getTaskVertices()) {
+
+ assertEquals(ExecutionState.CREATED, ev.getExecutionState());
+
+ LogicalSlot slot =
+ new TestingLogicalSlotBuilder()
+ .setTaskManagerGateway(taskManagerGateway)
+ .createTestingLogicalSlot();
+ ev.getCurrentExecutionAttempt()
+ .registerProducedPartitions(slot.getTaskManagerLocation(), true)
+ .get();
+ ev.deployToSlot(slot);
+ assertEquals(ExecutionState.DEPLOYING, ev.getExecutionState());
+
+ TaskDeploymentDescriptor tdd = tdds.take();
+ assertNotNull(tdd);
+
+ List<InputGateDeploymentDescriptor> igdds = tdd.getInputGates();
+ assertEquals(ev.getAllConsumedPartitionGroups().size(), igdds.size());
+
+ if (igdds.size() > 0) {
+ assertShuffleDescriptorsEqual(igdds.get(0), ev.getConsumedPartitionGroup(0));
+ }
+ }
+ }
+ }
+
+ private ExecutionGraph createAndSetupExecutionGraph(int numberOfVertices, int parallelism)
+ throws JobException, JobExecutionException {
+
+ final List<JobVertex> vertices = new ArrayList<>();
+
+ for (int i = 0; i < numberOfVertices; i++) {
+ JobVertex vertex = new JobVertex(String.format("v%d", i + 1), new JobVertexID());
+ vertex.setParallelism(parallelism);
+ vertex.setInvokableClass(BatchTask.class);
+ vertices.add(vertex);
+ }
+
+ for (int i = 1; i < numberOfVertices; i++) {
+ vertices.get(i)
+ .connectNewDataSetAsInput(
+ vertices.get(i - 1),
+ DistributionPattern.ALL_TO_ALL,
+ ResultPartitionType.BLOCKING);
+ }
+
+ final JobGraph jobGraph =
+ JobGraphTestUtils.batchJobGraph(vertices.toArray(new JobVertex[0]));
+
+ final DirectScheduledExecutorService executor = new DirectScheduledExecutorService();
+ final DefaultExecutionGraph eg =
+ TestingDefaultExecutionGraphBuilder.newBuilder()
+ .setJobGraph(jobGraph)
+ .setFutureExecutor(executor)
+ .setIoExecutor(executor)
+ .setBlobWriter(blobWriter)
+ .build();
+
+ eg.start(ComponentMainThreadExecutorServiceAdapter.forMainThread());
+
+ return eg;
+ }
+
+ private static void assertShuffleDescriptorsEqual(
Review comment:
Thank you for pointing this out. Resolved.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
+ if (blobServer != null) {
+ blobServer.close();
+ }
+ }
+
+ @Test
+ public void testDeployTasksWithMinimumSizeLimit() throws Exception {
Review comment:
Because in `DefaultExecutionGraphDeploymentTest#testBuildDeploymentDescriptor` it only deploys one task. It cannot test the scenario that the ShuffleDescriptor blob cache is replaced every time a new task is deployed since the size limit is 1. There's another option that we can modify `testBuildDeploymentDescriptor` to deploy all the tasks. WDYT?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ temporaryFolder.create();
+ }
+
+ @AfterClass
+ public static void cleanUp() throws IOException {
+ temporaryFolder.delete();
+ }
+
+ @Test
+ public void testSizeLimitTrackerStatusUpdate() {
+ BlobCacheSizeTracker tracker = new BlobCacheSizeTracker(5L);
+ List<JobID> jobIds = new ArrayList<>();
+ List<BlobKey> blobKeys = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ jobIds.add(new JobID());
+ blobKeys.add(BlobKey.createKey(BlobKey.BlobType.PERMANENT_BLOB));
+ }
+ for (int i = 0; i < 5; i++) {
+ tracker.track(jobIds.get(i), blobKeys.get(i), 1);
+ }
+ tracker.update(jobIds.get(1), blobKeys.get(1));
+ tracker.update(jobIds.get(2), blobKeys.get(2));
+
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = tracker.checkLimit(2);
+
+ assertThat(
+ blobsToDelete,
+ containsInAnyOrder(
+ Tuple2.of(jobIds.get(0), blobKeys.get(0)),
+ Tuple2.of(jobIds.get(3), blobKeys.get(3))));
+ }
+
+ @Test
+ public void testTrackPermanentBlobCache() throws Exception {
Review comment:
Moved to `PermanentBlobCacheSizeLimitTest`, which is a test for the feature.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -247,6 +404,7 @@ public void run() {
boolean success = false;
try {
+ blobCacheSizeTracker.unregisterJob(jobId);
Review comment:
I'm not sure about that. `PermanentBlobCache#releaseJob()` only minus a ref of the `jobRefCounter`. A job is truly released when the ref is zero and the time has passed the `cleanupInterval`. For example, if the TM has only one slot, the blob cache will be removed every time a task is finished.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
+ BlobKey blobKey,
+ File localFile,
+ Logger log,
+ @Nullable BlobStore blobStore)
+ throws IOException {
+
+ // Check the size limit and delete the files that exceeds the limit
+ final long sizeOfIncomingFile = incomingFile.length();
+ final List<Tuple2<JobID, BlobKey>> blobsToDelete =
+ blobCacheSizeTracker.checkLimit(sizeOfIncomingFile);
+
+ for (Tuple2<JobID, BlobKey> key : blobsToDelete) {
+ deleteFile(key.f0, key.f1);
+ blobCacheSizeTracker.untrack(key);
+ }
+
+ // Move the file and register it to the tracker
+ BlobUtils.moveTempFileToStore(incomingFile, jobId, blobKey, localFile, log, blobStore);
+ blobCacheSizeTracker.track(jobId, blobKey, localFile.length());
+ }
+
+ /**
+ * Delete the blob file with the given key.
+ *
+ * @param jobId ID of the job this blob belongs to (or <tt>null</tt> if job-unrelated)
+ * @param blobKey The key of the desired BLOB.
+ */
+ private void deleteFile(@Nullable JobID jobId, BlobKey blobKey) {
Review comment:
Done.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
Review comment:
Yes, you are right. Since we are not trying to hold WRITE/READ locks in the tracker any more, there will be no deadlocks here. And if `track/untrack/checkLimit` is not guarded by the WRITE/READ lock, there may be concurrent threads trying to check the limit in the middle of the `checkLimits` and `untrack`, making the track inconsistent.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
Review comment:
Thank you for pointing this out. Resolved.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
+ BlobKey blobKey,
+ File localFile,
+ Logger log,
+ @Nullable BlobStore blobStore)
+ throws IOException {
+
+ // Check the size limit and delete the files that exceeds the limit
+ final long sizeOfIncomingFile = incomingFile.length();
+ final List<Tuple2<JobID, BlobKey>> blobsToDelete =
+ blobCacheSizeTracker.checkLimit(sizeOfIncomingFile);
+
+ for (Tuple2<JobID, BlobKey> key : blobsToDelete) {
+ deleteFile(key.f0, key.f1);
+ blobCacheSizeTracker.untrack(key);
Review comment:
Since here we use a for-loop, there will be no infinite loop here. The only risk is that we will try to delete the blob every time a new blob inserts until the deletion is successful. I'll make `deleteFile` return the result of deletion and won't untrack the blob if deletion is failed.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
Review comment:
Thank you for pointing this out. I get to learn about how `@Rule` works.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ temporaryFolder.create();
+ }
+
+ @AfterClass
+ public static void cleanUp() throws IOException {
+ temporaryFolder.delete();
+ }
+
+ @Test
+ public void testSizeLimitTrackerStatusUpdate() {
+ BlobCacheSizeTracker tracker = new BlobCacheSizeTracker(5L);
+ List<JobID> jobIds = new ArrayList<>();
+ List<BlobKey> blobKeys = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ jobIds.add(new JobID());
+ blobKeys.add(BlobKey.createKey(BlobKey.BlobType.PERMANENT_BLOB));
+ }
+ for (int i = 0; i < 5; i++) {
+ tracker.track(jobIds.get(i), blobKeys.get(i), 1);
+ }
+ tracker.update(jobIds.get(1), blobKeys.get(1));
+ tracker.update(jobIds.get(2), blobKeys.get(2));
+
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = tracker.checkLimit(2);
+
+ assertThat(
+ blobsToDelete,
+ containsInAnyOrder(
+ Tuple2.of(jobIds.get(0), blobKeys.get(0)),
+ Tuple2.of(jobIds.get(3), blobKeys.get(3))));
+ }
+
+ @Test
+ public void testTrackPermanentBlobCache() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * blobSize)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][blobSize];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[blobSize];
+ random.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
+ readFileAndVerifyContent(cache, jobId[1], key[1], data[1]);
+
+ // Then get the fourth blob, make sure the third blob is replaced.
+ // Here we cannot get the first blob again, because for transient blob,
+ // BlobServer will delete the blob once it's transmitted.
+ key[3] = put(server, jobId[3], data[3], blobType);
+ readFileAndVerifyContent(cache, jobId[3], key[3], data[3]);
+ file[3] = getFile(cache, jobId[3], key[3]);
+
+ assertTrue(file[1].exists());
+ assertTrue(file[3].exists());
+ assertFalse(file[2].exists());
+ }
+ }
+
+ @Test
+ public void testGetPermanentCacheWithSizeLimitConcurrently() throws Exception {
Review comment:
Moved to `PermanentBlobCacheSizeLimitTest`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ untrack(Tuple2.of(jobId, blobKey));
+ }
+
+ /**
+ * Update the last used index for the BLOBs so that the tracker can easily find out the last
+ * recently used BLOBs.
+ */
+ public void update(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ synchronized (lock) {
+ caches.get(Tuple2.of(jobId, blobKey));
+ }
+ }
+
+ /** Unregister all the tracked BLOBs related to given job. */
+ public void unregisterJob(JobID jobId) {
+ checkNotNull(jobId);
+
+ synchronized (lock) {
+ for (BlobKey blobKey : blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>())) {
+ untrack(jobId, blobKey);
+ }
+ blobKeyByJob.remove(jobId);
+ }
+ }
+
+ @VisibleForTesting
+ Set<BlobKey> getBlobKeysByJobId(JobID jobId) {
Review comment:
It's not used in production. It's used in `BlobCacheCleanupTest#checkCacheSizeTracker`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
Review comment:
The comment has been improved slightly.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobService.java
##########
@@ -45,4 +46,21 @@
* @throws IOException if any other error occurs when retrieving the file
*/
File getFile(JobID jobId, PermanentBlobKey key) throws IOException;
+
+ /**
+ * Directly read the file associated with the provided job ID and blob key.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read when
Review comment:
Yes. I agree that the description here is not accurate. Maybe we should speak about the concurrency in the descriptions of `readFile` in `PermanentBlobCache` rather than here?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -102,6 +123,9 @@ public PermanentBlobCache(
this.cleanupInterval = blobClientConfig.getLong(BlobServerOptions.CLEANUP_INTERVAL) * 1000;
this.cleanupTimer.schedule(
new PermanentBlobCleanupTask(), cleanupInterval, cleanupInterval);
+
+ // Configure the size limit tracker
Review comment:
Agreed. Removed.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
Review comment:
Sorry for being careless. I've noticed that. Resolved.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
Review comment:
Yes. Sorry for being careless. I've removed it.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
Review comment:
For sanity I add a `checkState` here.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
Review comment:
Done.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ temporaryFolder.create();
+ }
+
+ @AfterClass
+ public static void cleanUp() throws IOException {
+ temporaryFolder.delete();
+ }
+
+ @Test
+ public void testSizeLimitTrackerStatusUpdate() {
Review comment:
I get it. I'll add more tests for functions of `BlobCacheSizeTracker`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
Review comment:
Thank you for pointing this out. I shouldn't ignore this. Resolved.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ untrack(Tuple2.of(jobId, blobKey));
+ }
+
+ /**
+ * Update the last used index for the BLOBs so that the tracker can easily find out the last
+ * recently used BLOBs.
+ */
+ public void update(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ synchronized (lock) {
+ caches.get(Tuple2.of(jobId, blobKey));
+ }
+ }
+
+ /** Unregister all the tracked BLOBs related to given job. */
+ public void unregisterJob(JobID jobId) {
Review comment:
Thank you for providing such a better name. I've thought about it for a long time. Resolved.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -90,6 +96,21 @@ public PermanentBlobCache(
@Nullable final InetSocketAddress serverAddress)
throws IOException {
+ this(
+ blobClientConfig,
+ blobView,
+ serverAddress,
+ new BlobCacheSizeTracker(MemorySize.ofMebiBytes(100).getBytes()));
Review comment:
Thank you for providing this suggestion. Done.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ untrack(Tuple2.of(jobId, blobKey));
+ }
+
+ /**
+ * Update the last used index for the BLOBs so that the tracker can easily find out the last
+ * recently used BLOBs.
+ */
+ public void update(@Nullable JobID jobId, BlobKey blobKey) {
Review comment:
Done.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
Review comment:
Alright. If duplicated, new comers will be ignored. Warning will be output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* b9d70021d437374b456556dce817d203bd4f21e2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070)
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 4175d2bca779b9a7f4dfeb9e9f5905777faea0e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 00805fc22fb911bbebe4efb0b5ffc4a29aeac25c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004)
* ec9fa78a8da0cbf99cb1a9a342275923176d3484 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031)
* 7ce8a08f3d996eee8c4ac7ac2de4618152099ac7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677225022
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ temporaryFolder.create();
+ }
+
+ @AfterClass
+ public static void cleanUp() throws IOException {
+ temporaryFolder.delete();
+ }
+
+ @Test
+ public void testSizeLimitTrackerStatusUpdate() {
Review comment:
Can be simplified to be `testUpdate `
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
Review comment:
Can be simplified to be:
```
@Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
```
And the `setUp()` and `cleanUp ()` will not be needed.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ temporaryFolder.create();
+ }
+
+ @AfterClass
+ public static void cleanUp() throws IOException {
+ temporaryFolder.delete();
+ }
+
+ @Test
+ public void testSizeLimitTrackerStatusUpdate() {
+ BlobCacheSizeTracker tracker = new BlobCacheSizeTracker(5L);
+ List<JobID> jobIds = new ArrayList<>();
+ List<BlobKey> blobKeys = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ jobIds.add(new JobID());
+ blobKeys.add(BlobKey.createKey(BlobKey.BlobType.PERMANENT_BLOB));
+ }
+ for (int i = 0; i < 5; i++) {
+ tracker.track(jobIds.get(i), blobKeys.get(i), 1);
+ }
+ tracker.update(jobIds.get(1), blobKeys.get(1));
+ tracker.update(jobIds.get(2), blobKeys.get(2));
+
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = tracker.checkLimit(2);
+
+ assertThat(
+ blobsToDelete,
+ containsInAnyOrder(
+ Tuple2.of(jobIds.get(0), blobKeys.get(0)),
+ Tuple2.of(jobIds.get(3), blobKeys.get(3))));
+ }
+
+ @Test
+ public void testTrackPermanentBlobCache() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * blobSize)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][blobSize];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[blobSize];
+ random.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
+ readFileAndVerifyContent(cache, jobId[1], key[1], data[1]);
+
+ // Then get the fourth blob, make sure the third blob is replaced.
+ // Here we cannot get the first blob again, because for transient blob,
+ // BlobServer will delete the blob once it's transmitted.
+ key[3] = put(server, jobId[3], data[3], blobType);
+ readFileAndVerifyContent(cache, jobId[3], key[3], data[3]);
+ file[3] = getFile(cache, jobId[3], key[3]);
+
+ assertTrue(file[1].exists());
+ assertTrue(file[3].exists());
+ assertFalse(file[2].exists());
+ }
+ }
+
+ @Test
+ public void testGetPermanentCacheWithSizeLimitConcurrently() throws Exception {
Review comment:
It is test of `PermanentBlobCache` and should not be in `BlobCacheSizeTrackerTest`.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.TestLogger;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private static final int blobSize = 10_000;
+
+ private static final Random random = new Random();
+
+ private static final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+ temporaryFolder.create();
+ }
+
+ @AfterClass
+ public static void cleanUp() throws IOException {
+ temporaryFolder.delete();
+ }
+
+ @Test
+ public void testSizeLimitTrackerStatusUpdate() {
+ BlobCacheSizeTracker tracker = new BlobCacheSizeTracker(5L);
+ List<JobID> jobIds = new ArrayList<>();
+ List<BlobKey> blobKeys = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ jobIds.add(new JobID());
+ blobKeys.add(BlobKey.createKey(BlobKey.BlobType.PERMANENT_BLOB));
+ }
+ for (int i = 0; i < 5; i++) {
+ tracker.track(jobIds.get(i), blobKeys.get(i), 1);
+ }
+ tracker.update(jobIds.get(1), blobKeys.get(1));
+ tracker.update(jobIds.get(2), blobKeys.get(2));
+
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = tracker.checkLimit(2);
+
+ assertThat(
+ blobsToDelete,
+ containsInAnyOrder(
+ Tuple2.of(jobIds.get(0), blobKeys.get(0)),
+ Tuple2.of(jobIds.get(3), blobKeys.get(3))));
+ }
+
+ @Test
+ public void testTrackPermanentBlobCache() throws Exception {
Review comment:
It is test of `PermanentBlobCache` and should not be in `BlobCacheSizeTrackerTest`.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ untrack(Tuple2.of(jobId, blobKey));
+ }
+
+ /**
+ * Update the last used index for the BLOBs so that the tracker can easily find out the last
+ * recently used BLOBs.
+ */
+ public void update(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ synchronized (lock) {
+ caches.get(Tuple2.of(jobId, blobKey));
+ }
+ }
+
+ /** Unregister all the tracked BLOBs related to given job. */
+ public void unregisterJob(JobID jobId) {
+ checkNotNull(jobId);
+
+ synchronized (lock) {
+ for (BlobKey blobKey : blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>())) {
+ untrack(jobId, blobKey);
+ }
+ blobKeyByJob.remove(jobId);
+ }
+ }
+
+ @VisibleForTesting
+ Set<BlobKey> getBlobKeysByJobId(JobID jobId) {
Review comment:
Seems this method is never used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 097a051078c80ca3f757f8734d484f84f08e7b21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969)
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 00805fc22fb911bbebe4efb0b5ffc4a29aeac25c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677445872
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobService.java
##########
@@ -45,4 +46,21 @@
* @throws IOException if any other error occurs when retrieving the file
*/
File getFile(JobID jobId, PermanentBlobKey key) throws IOException;
+
+ /**
+ * Directly read the file associated with the provided job ID and blob key.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read when
Review comment:
It's better to illustrate here that why we need to add this interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 097a051078c80ca3f757f8734d484f84f08e7b21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c22129c7d12014419bc3a8d66934f97fb1df6a53 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914)
* 097a051078c80ca3f757f8734d484f84f08e7b21 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677298097
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
Review comment:
Alright. If duplicated, new comers will be ignored. Warning will output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678116676
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
+ readFileAndVerifyContent(cache, jobId[1], key[1], data[1]);
+
+ // Then get the fourth blob, make sure the third blob is replaced.
+ // Here we cannot get the first blob again, because for transient blob,
+ // BlobServer will delete the blob once it's transmitted.
+ key[3] = put(server, jobId[3], data[3], blobType);
+ readFileAndVerifyContent(cache, jobId[3], key[3], data[3]);
+ file[3] = getFile(cache, jobId[3], key[3]);
+
+ assertTrue(file[1].exists());
+ assertTrue(file[3].exists());
+ assertFalse(file[2].exists());
+ }
+ }
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcessConcurrently() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ final int concurrency = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(concurrency);
Review comment:
Thank you for pointing this out. I should've notice issues like this. This test has been refactored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7095135b7315584613d01410819d8e3b78446dac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836)
* c22129c7d12014419bc3a8d66934f97fb1df6a53 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51db50340f762127f3c3c83dc86f0ac08f0877d9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r671102572
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrency modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link AbstractBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ private final long capacity;
Review comment:
name it as `sizeLimit` would be easier for understanding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911cd22a5de05ab8603c7f75fc58b22b96eef2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124",
"triggerID" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21133",
"triggerID" : "888721980",
"triggerType" : "MANUAL"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124",
"triggerID" : "888721980",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 01f718e46e76efe7f74fa927a82a5329f5ef012b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21133) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678055864
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
+ BlobKey blobKey,
+ File localFile,
+ Logger log,
+ @Nullable BlobStore blobStore)
+ throws IOException {
+
+ // Check the size limit and delete the files that exceeds the limit
+ final long sizeOfIncomingFile = incomingFile.length();
+ final List<Tuple2<JobID, BlobKey>> blobsToDelete =
+ blobCacheSizeTracker.checkLimit(sizeOfIncomingFile);
+
+ for (Tuple2<JobID, BlobKey> key : blobsToDelete) {
+ deleteFile(key.f0, key.f1);
+ blobCacheSizeTracker.untrack(key);
Review comment:
I mean I think it's not "risk is that we will try to delete the blob every time a new blob inserts until the deletion is successful". It's good to log warnings and retry deletions in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51db50340f762127f3c3c83dc86f0ac08f0877d9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479)
* 0be481b90be72b85b8a294d9e14d08cad49ee40d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* b9d70021d437374b456556dce817d203bd4f21e2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070)
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 4175d2bca779b9a7f4dfeb9e9f5905777faea0e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096)
* ff39fb582bd29f5151b2a1625a3cf651c564d1f3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677938458
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
+ if (blobServer != null) {
+ blobServer.close();
+ }
+ }
+
+ @Test
+ public void testDeployTasksWithMinimumSizeLimit() throws Exception {
Review comment:
Ok. Then how about changing the test name to `testDeployMultipleTasksWithBlobCacheSizeLimit`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 00805fc22fb911bbebe4efb0b5ffc4a29aeac25c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004)
* ec9fa78a8da0cbf99cb1a9a342275923176d3484 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031)
* 7ce8a08f3d996eee8c4ac7ac2de4618152099ac7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 7ce8a08f3d996eee8c4ac7ac2de4618152099ac7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r673859613
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/BlobServerOptions.java
##########
@@ -118,4 +118,10 @@
key("blob.client.connect.timeout")
.defaultValue(0)
.withDescription("The connection timeout in milliseconds for the blob client.");
+
+ /** The maximum size limit of blob cache on TaskExecutor. */
+ public static final ConfigOption<Long> BLOB_CACHE_SIZE_LIMIT =
+ key("blob.cache.size.limit")
+ .defaultValue(1_024 * 1_024 * 1_024 * 10L)
+ .withDescription("The maximum size limit of blob cache for each TaskExecutor.");
Review comment:
> the description needs to clearly state how it affects permanent blob and transient blob size limit.
I've updated the descriptions here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 7ce8a08f3d996eee8c4ac7ac2de4618152099ac7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038)
* b9d70021d437374b456556dce817d203bd4f21e2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911cd22a5de05ab8603c7f75fc58b22b96eef2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798)
* 7095135b7315584613d01410819d8e3b78446dac UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678406858
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are intended to be stored and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final Random RANDOM = new Random();
+
+ private static final BlobKey.BlobType BLOB_TYPE = BlobKey.BlobType.PERMANENT_BLOB;
+ private static final int BLOB_SIZE = 10_000;
+ // The size limit is the size of 2 blobs
+ private static final int MAX_NUM_OF_ACCEPTED_BLOBS = 2;
+ private static final int TOTAL_NUM_OF_BLOBS = 3;
+
+ private static final Configuration CONFIG = new Configuration();
Review comment:
I think it's better not to make `CONFIG` and `TEMPORARY_FOLDER` static and shared between tests.
It's safer to initialize them before each test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678201387
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are intended to be stored and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final Random RANDOM = new Random();
+
+ private static final BlobKey.BlobType BLOB_TYPE = BlobKey.BlobType.PERMANENT_BLOB;
+ private static final int BLOB_SIZE = 10_000;
+ // The size limit is the size of 2 blobs
+ private static final int MAX_NUM_OF_ACCEPTED_BLOBS = 2;
+ private static final int TOTAL_NUM_OF_BLOBS = 3;
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcessSequentially() throws Exception {
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
Review comment:
Looks to me the `config` can be a class wide field.
And seems `initBlobCacheServiceWithSizeLimit()` can be param-less.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124",
"triggerID" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 01f718e46e76efe7f74fa927a82a5329f5ef012b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911cd22a5de05ab8603c7f75fc58b22b96eef2fa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798)
* 7095135b7315584613d01410819d8e3b78446dac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677941649
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
Review comment:
We should also add `checkNotNull` for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 097a051078c80ca3f757f8734d484f84f08e7b21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969)
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 00805fc22fb911bbebe4efb0b5ffc4a29aeac25c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677984188
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -99,44 +101,56 @@ public BlobCacheSizeTracker(long sizeLimit) {
}
/** Register the target file to the tracker. */
- public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ public void track(JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(jobId);
checkNotNull(blobKey);
checkArgument(size >= 0);
synchronized (lock) {
- caches.put(Tuple2.of(jobId, blobKey), size);
- if (jobId != null) {
+ if (caches.putIfAbsent(Tuple2.of(jobId, blobKey), size) == null) {
blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
- }
- total += size;
- if (total > sizeLimit) {
+
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds "
+ + "the limit. Limit = [{}], Current: [{}], "
+ + "The size of next ShuffleDescriptors: [{}].",
+ sizeLimit,
+ total,
+ size);
+ }
+ } else {
LOG.warn(
Review comment:
I think this warning shouldn't happen frequently. In `BlobUtil.moveTempFileToStore`, if the blob key duplicates, a warning will be outputted, too. Furthermore, the previous warning ("The overall size of ShuffleDescriptors in PermanentBlobCache exceeds ...") should be warning, too. The overall size exceeds if and only if the size of blob exceeds 100MiB. In this scenario the user should be warned about it.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobService.java
##########
@@ -50,13 +50,20 @@
/**
* Returns the content of the file for the BLOB with the provided job ID the blob key.
*
+ * <p>Compared to {@code getFile}, {@code readFile} will attempt to read the entire file after
+ * retrieving it. If file reading and file retrieving is done in the same WRITE lock, it can
+ * avoid the scenario that the path to the file is deleted concurrently by other threads at the
+ * same time as the file is retrieved and read.
Review comment:
Thank you for pointing this out. Resolved.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -99,44 +101,56 @@ public BlobCacheSizeTracker(long sizeLimit) {
}
/** Register the target file to the tracker. */
- public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ public void track(JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(jobId);
checkNotNull(blobKey);
checkArgument(size >= 0);
synchronized (lock) {
- caches.put(Tuple2.of(jobId, blobKey), size);
- if (jobId != null) {
+ if (caches.putIfAbsent(Tuple2.of(jobId, blobKey), size) == null) {
blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
- }
- total += size;
- if (total > sizeLimit) {
+
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds "
+ + "the limit. Limit = [{}], Current: [{}], "
+ + "The size of next ShuffleDescriptors: [{}].",
Review comment:
Resolved.
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -99,44 +101,56 @@ public BlobCacheSizeTracker(long sizeLimit) {
}
/** Register the target file to the tracker. */
- public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ public void track(JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(jobId);
checkNotNull(blobKey);
checkArgument(size >= 0);
synchronized (lock) {
- caches.put(Tuple2.of(jobId, blobKey), size);
- if (jobId != null) {
+ if (caches.putIfAbsent(Tuple2.of(jobId, blobKey), size) == null) {
blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
- }
- total += size;
- if (total > sizeLimit) {
+
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds "
Review comment:
Thank you for pointing this out. Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677979942
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
+ BlobKey blobKey,
+ File localFile,
+ Logger log,
+ @Nullable BlobStore blobStore)
+ throws IOException {
+
+ // Check the size limit and delete the files that exceeds the limit
+ final long sizeOfIncomingFile = incomingFile.length();
+ final List<Tuple2<JobID, BlobKey>> blobsToDelete =
+ blobCacheSizeTracker.checkLimit(sizeOfIncomingFile);
+
+ for (Tuple2<JobID, BlobKey> key : blobsToDelete) {
+ deleteFile(key.f0, key.f1);
+ blobCacheSizeTracker.untrack(key);
Review comment:
I'm not sure the meaning of exposing the problem. If the deletion is failed, `deleteFile` will output a warning log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0be481b90be72b85b8a294d9e14d08cad49ee40d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780)
* 911cd22a5de05ab8603c7f75fc58b22b96eef2fa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk closed pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk closed pull request #16498:
URL: https://github.com/apache/flink/pull/16498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124",
"triggerID" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* ff39fb582bd29f5151b2a1625a3cf651c564d1f3 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112)
* 01f718e46e76efe7f74fa927a82a5329f5ef012b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 4175d2bca779b9a7f4dfeb9e9f5905777faea0e3 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096)
* ff39fb582bd29f5151b2a1625a3cf651c564d1f3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880494243
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 01f718e46e76efe7f74fa927a82a5329f5ef012b (Thu Sep 23 18:09:31 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677982428
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraphDeploymentWithSizeLimitTest.java
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.executiongraph;
+
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.JobException;
+import org.apache.flink.runtime.blob.BlobCacheSizeTracker;
+import org.apache.flink.runtime.blob.BlobServer;
+import org.apache.flink.runtime.blob.PermanentBlobCache;
+import org.apache.flink.runtime.blob.VoidBlobStore;
+import org.apache.flink.runtime.client.JobExecutionException;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutorServiceAdapter;
+import org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.utils.SimpleAckingTaskManagerGateway;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.IntermediateResultPartitionID;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobGraphTestUtils;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.operators.BatchTask;
+import org.apache.flink.runtime.scheduler.strategy.ConsumedPartitionGroup;
+import org.apache.flink.runtime.testutils.DirectScheduledExecutorService;
+import org.apache.flink.util.function.FunctionUtils;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+
+public class DefaultExecutionGraphDeploymentWithSizeLimitTest
+ extends DefaultExecutionGraphDeploymentWithBlobCacheTest {
+
+ @Before
+ @Override
+ public void setupBlobServer() throws IOException {
+ Configuration config = new Configuration();
+ // Always offload the serialized JobInformation, TaskInformation and cached
+ // ShuffleDescriptors
+ config.setInteger(BlobServerOptions.OFFLOAD_MINSIZE, 0);
+ blobServer = new BlobServer(config, new VoidBlobStore());
+ blobServer.start();
+ blobWriter = blobServer;
+
+ InetSocketAddress serverAddress = new InetSocketAddress("localhost", blobServer.getPort());
+ // Set the size limit of the blob cache to 1
+ BlobCacheSizeTracker blobCacheSizeTracker = new BlobCacheSizeTracker(1L);
+ blobCache =
+ new PermanentBlobCache(
+ config, new VoidBlobStore(), serverAddress, blobCacheSizeTracker);
+ }
+
+ @After
+ @Override
+ public void shutdownBlobServer() throws IOException {
+ if (blobServer != null) {
+ blobServer.close();
+ }
+ }
+
+ @Test
+ public void testDeployTasksWithMinimumSizeLimit() throws Exception {
Review comment:
Sure. Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678115943
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.blob.BlobKey.BlobType;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private BlobCacheSizeTracker tracker;
+ private JobID jobId;
+ private BlobKey blobKey;
+
+ @Before
+ public void setup() {
+ tracker = new BlobCacheSizeTracker(5L);
+ jobId = new JobID();
+ blobKey = BlobKey.createKey(BlobType.PERMANENT_BLOB);
+
+ tracker.track(jobId, blobKey, 3L);
+ }
+
+ @Test
+ public void testCheckLimit() {
+ List<Tuple2<JobID, BlobKey>> keys = tracker.checkLimit(3L);
+
+ assertEquals(1, keys.size());
+ assertEquals(jobId, keys.get(0).f0);
+ assertEquals(blobKey, keys.get(0).f1);
+ }
+
+ @Test
+ public void testZero() {
Review comment:
I feel so awful for these low-level issues. I could do better but I didn't. I should've examined and got them fixed in the early stage. I'll keep in mind and try to do better next time. Thank you so much for being patient and helping me improve the skill. BTW, this unit test has been improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51db50340f762127f3c3c83dc86f0ac08f0877d9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124",
"triggerID" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21133",
"triggerID" : "888721980",
"triggerType" : "MANUAL"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124",
"triggerID" : "888721980",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 01f718e46e76efe7f74fa927a82a5329f5ef012b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21133) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21124)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 00805fc22fb911bbebe4efb0b5ffc4a29aeac25c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004)
* ec9fa78a8da0cbf99cb1a9a342275923176d3484 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677246841
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
Review comment:
`jobId` should be non-null for permanent blob cache
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
+ BlobKey blobKey,
+ File localFile,
+ Logger log,
+ @Nullable BlobStore blobStore)
+ throws IOException {
+
+ // Check the size limit and delete the files that exceeds the limit
+ final long sizeOfIncomingFile = incomingFile.length();
+ final List<Tuple2<JobID, BlobKey>> blobsToDelete =
+ blobCacheSizeTracker.checkLimit(sizeOfIncomingFile);
+
+ for (Tuple2<JobID, BlobKey> key : blobsToDelete) {
+ deleteFile(key.f0, key.f1);
+ blobCacheSizeTracker.untrack(key);
+ }
+
+ // Move the file and register it to the tracker
+ BlobUtils.moveTempFileToStore(incomingFile, jobId, blobKey, localFile, log, blobStore);
+ blobCacheSizeTracker.track(jobId, blobKey, localFile.length());
+ }
+
+ /**
+ * Delete the blob file with the given key.
+ *
+ * @param jobId ID of the job this blob belongs to (or <tt>null</tt> if job-unrelated)
+ * @param blobKey The key of the desired BLOB.
+ */
+ private void deleteFile(@Nullable JobID jobId, BlobKey blobKey) {
Review comment:
`jobId` should be non-null for permanent blob cache
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ untrack(Tuple2.of(jobId, blobKey));
+ }
+
+ /**
+ * Update the last used index for the BLOBs so that the tracker can easily find out the last
+ * recently used BLOBs.
+ */
+ public void update(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ synchronized (lock) {
+ caches.get(Tuple2.of(jobId, blobKey));
+ }
+ }
+
+ /** Unregister all the tracked BLOBs related to given job. */
+ public void unregisterJob(JobID jobId) {
Review comment:
I prefer the name to be `untrackAll` for easier understanding and also because there is no `registerJob()`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
Review comment:
`jobId` should be non-null for permanent blob cache
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
Review comment:
`jobId` should be non-null for permanent blob cache
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -247,6 +404,7 @@ public void run() {
boolean success = false;
try {
+ blobCacheSizeTracker.unregisterJob(jobId);
Review comment:
Can we do it in `PermanentBlobCache#releaseJob()`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
Review comment:
`put` -> `putIfAbsent` and increate total only if it is newly added
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
Review comment:
The size is never expected to be lesser than 0. We should check it instead of filter it.
It's also Ok to not check it because it was already checked on `track`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
+ checkNotNull(blobKey);
+
+ untrack(Tuple2.of(jobId, blobKey));
+ }
+
+ /**
+ * Update the last used index for the BLOBs so that the tracker can easily find out the last
+ * recently used BLOBs.
+ */
+ public void update(@Nullable JobID jobId, BlobKey blobKey) {
Review comment:
`jobId` should be non-null for permanent blob cache
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * BlobCacheSizeTracker uses {@link LinkedHashMap} to maintain the LRU order for the files in the
+ * cache. When new files are intended to be put into cache, {@code checkLimit} is called to query
+ * the files should be removed. This tracker maintains a lock to avoid concurrent modification. To
+ * avoid the deadlock, make sure that hold the READ/WRITE lock in {@link PermanentBlobCache} first
+ * and then hold the lock here.
+ */
+public class BlobCacheSizeTracker {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BlobCacheSizeTracker.class);
+
+ private static final int INITIAL_SIZE = 10_000;
+
+ private final Object lock = new Object();
+
+ protected final long sizeLimit;
+
+ @GuardedBy("lock")
+ private long total;
+
+ @GuardedBy("lock")
+ private final LinkedHashMap<Tuple2<JobID, BlobKey>, Long> caches;
+
+ @GuardedBy("lock")
+ protected final HashMap<JobID, Set<BlobKey>> blobKeyByJob;
+
+ public BlobCacheSizeTracker(long sizeLimit) {
+ this.sizeLimit = sizeLimit;
+ this.total = 0L;
+ this.caches = new LinkedHashMap<>(INITIAL_SIZE, 0.75F, true);
+ this.blobKeyByJob = new HashMap<>();
+ }
+
+ /**
+ * Check the size limit and return the files to delete.
+ *
+ * @param size size of the target file intended to put into cache
+ * @return list of files to delete before saving the target file
+ */
+ public List<Tuple2<JobID, BlobKey>> checkLimit(long size) {
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ List<Tuple2<JobID, BlobKey>> blobsToDelete = new ArrayList<>();
+
+ long current = total;
+
+ for (Map.Entry<Tuple2<JobID, BlobKey>, Long> entry : caches.entrySet()) {
+ if (current + size > sizeLimit) {
+ blobsToDelete.add(entry.getKey());
+ current -= entry.getValue();
+ }
+ }
+
+ return blobsToDelete;
+ }
+ }
+
+ /** Register the target file to the tracker. */
+ public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(blobKey);
+ checkArgument(size >= 0);
+
+ synchronized (lock) {
+ caches.put(Tuple2.of(jobId, blobKey), size);
+ if (jobId != null) {
+ blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
+ }
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds the limit. "
+ + "Limit: {}, current: {}. The size of next ShuffleDescriptors: {}.",
+ sizeLimit,
+ total,
+ size);
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(Tuple2<JobID, BlobKey> key) {
+ checkNotNull(key);
+
+ synchronized (lock) {
+ if (key.f0 != null) {
+ blobKeyByJob.computeIfAbsent(key.f0, ignore -> new HashSet<>()).remove(key.f1);
+ }
+ Long size = caches.remove(key);
+ if (size != null && size >= 0) {
+ total -= size;
+ }
+ }
+ }
+
+ /** Remove the BLOB from the tracker. */
+ public void untrack(@Nullable JobID jobId, BlobKey blobKey) {
Review comment:
It can be private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-888821366
> Thanks for addressing all the comments @Thesharing
> The change now looks good to me. Will merge it once CI gives green.
Thank you for your reviewing, @zhuzhurk. It helps me a lot. The CI seems to be passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677165668
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobService.java
##########
@@ -45,4 +46,21 @@
* @throws IOException if any other error occurs when retrieving the file
*/
File getFile(JobID jobId, PermanentBlobKey key) throws IOException;
+
+ /**
+ * Directly read the file associated with the provided job ID and blob key.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read when
Review comment:
>>> {@code readFile} makes sure that ..
I think it's not very accurate because `readFile` does not guarantee that. However, it provides the possibility to achieve that, if the file reading and file retrieving happens in the same write lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7095135b7315584613d01410819d8e3b78446dac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836)
* c22129c7d12014419bc3a8d66934f97fb1df6a53 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0be481b90be72b85b8a294d9e14d08cad49ee40d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677943607
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -99,44 +101,56 @@ public BlobCacheSizeTracker(long sizeLimit) {
}
/** Register the target file to the tracker. */
- public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ public void track(JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(jobId);
checkNotNull(blobKey);
checkArgument(size >= 0);
synchronized (lock) {
- caches.put(Tuple2.of(jobId, blobKey), size);
- if (jobId != null) {
+ if (caches.putIfAbsent(Tuple2.of(jobId, blobKey), size) == null) {
blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
- }
- total += size;
- if (total > sizeLimit) {
+
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds "
Review comment:
`ShuffleDescriptors` -> `blob`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -99,44 +101,56 @@ public BlobCacheSizeTracker(long sizeLimit) {
}
/** Register the target file to the tracker. */
- public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ public void track(JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(jobId);
checkNotNull(blobKey);
checkArgument(size >= 0);
synchronized (lock) {
- caches.put(Tuple2.of(jobId, blobKey), size);
- if (jobId != null) {
+ if (caches.putIfAbsent(Tuple2.of(jobId, blobKey), size) == null) {
blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
- }
- total += size;
- if (total > sizeLimit) {
+
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds "
+ + "the limit. Limit = [{}], Current: [{}], "
+ + "The size of next ShuffleDescriptors: [{}].",
Review comment:
`ShuffleDescriptors` -> `blob`
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCacheSizeTracker.java
##########
@@ -99,44 +101,56 @@ public BlobCacheSizeTracker(long sizeLimit) {
}
/** Register the target file to the tracker. */
- public void track(@Nullable JobID jobId, BlobKey blobKey, long size) {
+ public void track(JobID jobId, BlobKey blobKey, long size) {
+ checkNotNull(jobId);
checkNotNull(blobKey);
checkArgument(size >= 0);
synchronized (lock) {
- caches.put(Tuple2.of(jobId, blobKey), size);
- if (jobId != null) {
+ if (caches.putIfAbsent(Tuple2.of(jobId, blobKey), size) == null) {
blobKeyByJob.computeIfAbsent(jobId, ignore -> new HashSet<>()).add(blobKey);
- }
- total += size;
- if (total > sizeLimit) {
+
+ total += size;
+ if (total > sizeLimit) {
+ LOG.warn(
+ "The overall size of ShuffleDescriptors in PermanentBlobCache exceeds "
+ + "the limit. Limit = [{}], Current: [{}], "
+ + "The size of next ShuffleDescriptors: [{}].",
+ sizeLimit,
+ total,
+ size);
+ }
+ } else {
LOG.warn(
Review comment:
Is it possible to happen frequently. If yes, I prefer to make it a debug log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 00805fc22fb911bbebe4efb0b5ffc4a29aeac25c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004)
* ec9fa78a8da0cbf99cb1a9a342275923176d3484 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51db50340f762127f3c3c83dc86f0ac08f0877d9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479)
* 0be481b90be72b85b8a294d9e14d08cad49ee40d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677533136
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobService.java
##########
@@ -45,4 +46,21 @@
* @throws IOException if any other error occurs when retrieving the file
*/
File getFile(JobID jobId, PermanentBlobKey key) throws IOException;
+
+ /**
+ * Directly read the file associated with the provided job ID and blob key.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read when
Review comment:
> It's better to illustrate here that why we need to add this interface.
More detailed comments are added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c22129c7d12014419bc3a8d66934f97fb1df6a53 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* 00805fc22fb911bbebe4efb0b5ffc4a29aeac25c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 097a051078c80ca3f757f8734d484f84f08e7b21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969)
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677940469
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -187,6 +211,139 @@ public File getFile(JobID jobId, PermanentBlobKey key) throws IOException {
return getFileInternal(jobId, key);
}
+ /**
+ * Returns the content of the file for the BLOB with the provided job ID the blob key.
+ *
+ * <p>The method will first attempt to serve the BLOB from the local cache. If the BLOB is not
+ * in the cache, the method will try to download it from the HA store, or directly from the
+ * {@link BlobServer}.
+ *
+ * <p>Compared to {@code getFile}, {@code readFile} makes sure that the file is fully read in
+ * the same write lock as the file is accessed. This avoids the scenario that the path is
+ * returned as the file is deleted concurrently by other threads.
+ *
+ * @param jobId ID of the job this blob belongs to
+ * @param blobKey BLOB key associated with the requested file
+ * @return The content of the BLOB.
+ * @throws java.io.FileNotFoundException if the BLOB does not exist;
+ * @throws IOException if any other error occurs when retrieving the file.
+ */
+ @Override
+ public byte[] readFile(JobID jobId, PermanentBlobKey blobKey) throws IOException {
+ checkNotNull(jobId);
+ checkNotNull(blobKey);
+
+ final File localFile = BlobUtils.getStorageLocation(storageDir, jobId, blobKey);
+ readWriteLock.readLock().lock();
+
+ try {
+ if (localFile.exists()) {
+ blobCacheSizeTracker.update(jobId, blobKey);
+ return FileUtils.readAllBytes(localFile.toPath());
+ }
+ } finally {
+ readWriteLock.readLock().unlock();
+ }
+
+ // first try the distributed blob store (if available)
+ // use a temporary file (thread-safe without locking)
+ File incomingFile = createTemporaryFilename();
+ try {
+ try {
+ if (blobView.get(jobId, blobKey, incomingFile)) {
+ // now move the temp file to our local cache atomically
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+ } catch (Exception e) {
+ log.info(
+ "Failed to copy from blob store. Downloading from BLOB server instead.", e);
+ }
+
+ final InetSocketAddress currentServerAddress = serverAddress;
+
+ if (currentServerAddress != null) {
+ // fallback: download from the BlobServer
+ BlobClient.downloadFromBlobServer(
+ jobId,
+ blobKey,
+ incomingFile,
+ currentServerAddress,
+ blobClientConfig,
+ numFetchRetries);
+
+ readWriteLock.writeLock().lock();
+ try {
+ checkLimitAndMoveFile(incomingFile, jobId, blobKey, localFile, log, null);
+ return FileUtils.readAllBytes(localFile.toPath());
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ } else {
+ throw new IOException(
+ "Cannot download from BlobServer, because the server address is unknown.");
+ }
+
+ } finally {
+ // delete incomingFile from a failed download
+ if (!incomingFile.delete() && incomingFile.exists()) {
+ log.warn(
+ "Could not delete the staging file {} for blob key {} and job {}.",
+ incomingFile,
+ blobKey,
+ jobId);
+ }
+ }
+ }
+
+ private void checkLimitAndMoveFile(
+ File incomingFile,
+ @Nullable JobID jobId,
+ BlobKey blobKey,
+ File localFile,
+ Logger log,
+ @Nullable BlobStore blobStore)
+ throws IOException {
+
+ // Check the size limit and delete the files that exceeds the limit
+ final long sizeOfIncomingFile = incomingFile.length();
+ final List<Tuple2<JobID, BlobKey>> blobsToDelete =
+ blobCacheSizeTracker.checkLimit(sizeOfIncomingFile);
+
+ for (Tuple2<JobID, BlobKey> key : blobsToDelete) {
+ deleteFile(key.f0, key.f1);
+ blobCacheSizeTracker.untrack(key);
Review comment:
>>> The only risk is that we will try to delete the blob every time a new blob inserts until the deletion is successful
I think it's better to have to expose the problem. And it also enables retry of failed deletions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677928575
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobService.java
##########
@@ -50,13 +50,20 @@
/**
* Returns the content of the file for the BLOB with the provided job ID the blob key.
*
+ * <p>Compared to {@code getFile}, {@code readFile} will attempt to read the entire file after
+ * retrieving it. If file reading and file retrieving is done in the same WRITE lock, it can
+ * avoid the scenario that the path to the file is deleted concurrently by other threads at the
+ * same time as the file is retrieved and read.
Review comment:
maybe "at the same time as the file is retrieved and read" -> "when the file is retrieved but not read yet"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* b9d70021d437374b456556dce817d203bd4f21e2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070)
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r671093353
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/AbstractBlobCache.java
##########
@@ -249,6 +278,28 @@ public void close() throws IOException {
}
}
+ /**
+ * Delete the blob file with the given key.
+ *
+ * @param jobId ID of the job this blob belongs to (or <tt>null</tt> if job-unrelated)
+ * @param blobKey The key of the desired BLOB.
+ * @return file successfully deleted or not.
+ */
+ public boolean deleteFileUnsafely(@Nullable JobID jobId, BlobKey blobKey) {
+ final File localFile =
+ new File(
+ BlobUtils.getStorageLocationPath(
+ storageDir.getAbsolutePath(), jobId, blobKey));
+ if (!localFile.delete() && localFile.exists()) {
+ log.warn(
+ "Failed to delete locally cached BLOB {} at {}",
+ blobKey,
+ localFile.getAbsolutePath());
+ return false;
Review comment:
> Why do we have a return value but add no handling for it?
Since the failure of deleting files is tolerable and the log has been outputted here, we can remove the return value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677982673
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.blob.BlobKey.BlobType;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private BlobCacheSizeTracker tracker;
+ private JobID jobId;
+ private BlobKey blobKey;
+
+ @Before
+ public void setup() {
+ tracker = new BlobCacheSizeTracker(5L);
+ jobId = new JobID();
+ blobKey = BlobKey.createKey(BlobType.PERMANENT_BLOB);
+
+ tracker.track(jobId, blobKey, 3L);
+ }
+
+ @Test
+ public void testCheckLimit() {
+ List<Tuple2<JobID, BlobKey>> keys = tracker.checkLimit(3L);
+
+ assertEquals(1, keys.size());
+ assertEquals(jobId, keys.get(0).f0);
+ assertEquals(blobKey, keys.get(0).f1);
+ }
+
+ @Test
+ public void testZero() {
Review comment:
The name is confusing. And I don't think this is expected to happen in production.
Maybe it should be `testCheckNonExceededLimit` and check a small size.
And I really hope this kind of changes can be examined and polished in ahead.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheSizeTrackerTest.java
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.blob.BlobKey.BlobType;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+/** Tests for {@link BlobCacheSizeTracker}. */
+public class BlobCacheSizeTrackerTest extends TestLogger {
+
+ private BlobCacheSizeTracker tracker;
+ private JobID jobId;
+ private BlobKey blobKey;
+
+ @Before
+ public void setup() {
+ tracker = new BlobCacheSizeTracker(5L);
+ jobId = new JobID();
+ blobKey = BlobKey.createKey(BlobType.PERMANENT_BLOB);
+
+ tracker.track(jobId, blobKey, 3L);
+ }
+
+ @Test
+ public void testCheckLimit() {
+ List<Tuple2<JobID, BlobKey>> keys = tracker.checkLimit(3L);
+
+ assertEquals(1, keys.size());
+ assertEquals(jobId, keys.get(0).f0);
+ assertEquals(blobKey, keys.get(0).f1);
+ }
+
+ @Test
+ public void testZero() {
+ List<Tuple2<JobID, BlobKey>> keys = tracker.checkLimit(0L);
+
+ assertEquals(0, keys.size());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testCheckNegative() {
+ BlobCacheSizeTracker tracker = new BlobCacheSizeTracker(5L);
Review comment:
Why do we need to create a new tracker?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
+ readFileAndVerifyContent(cache, jobId[1], key[1], data[1]);
+
+ // Then get the fourth blob, make sure the third blob is replaced.
+ // Here we cannot get the first blob again, because for transient blob,
+ // BlobServer will delete the blob once it's transmitted.
+ key[3] = put(server, jobId[3], data[3], blobType);
+ readFileAndVerifyContent(cache, jobId[3], key[3], data[3]);
+ file[3] = getFile(cache, jobId[3], key[3]);
+
+ assertTrue(file[1].exists());
+ assertTrue(file[3].exists());
+ assertFalse(file[2].exists());
+ }
+ }
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcessConcurrently() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ final int concurrency = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(concurrency);
+
+ // The size limit is the size of 2 blobs
+ final int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[concurrency];
+ BlobKey[] key = new BlobKey[concurrency];
+ File[] file = new File[concurrency];
+
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ // Put all the BLOBs into the blob server one by one
+ for (int i = 0; i < concurrency; i++) {
+ jobId[i] = new JobID();
+
+ key[i] = put(server, jobId[i], blobData, blobType);
+ assertNotNull(key[i]);
+ }
+
+ List<CompletableFuture<Void>> futures = new ArrayList<>(concurrency);
+
+ // Get and visit the blob cache concurrently
+ for (int i = 0; i < concurrency; i++) {
+ int idx = i;
+ CompletableFuture<Void> future =
+ CompletableFuture.supplyAsync(
+ () -> {
+ try {
+ file[idx] = getFile(cache, jobId[idx], key[idx]);
+ readFileAndVerifyContent(
+ cache, jobId[idx], key[idx], blobData);
+ return null;
+ } catch (IOException e) {
+ throw new CompletionException(e);
+ }
+ },
+ executor);
+
+ futures.add(future);
+ }
+
+ CompletableFuture<Void> conjunctFuture = FutureUtils.waitForAll(futures);
+ conjunctFuture.get();
+
+ // Check how many blob caches exists
+ int exists = 0, nonExists = 0;
+ for (int i = 0; i < concurrency; i++) {
+ if (file[i].exists()) {
+ exists++;
+ } else {
+ nonExists++;
+ }
+ }
+ assertEquals(sizeLimit, exists);
+ assertEquals(concurrency - sizeLimit, nonExists);
+ }
+ }
+
+ private static BlobCacheService initBlobCacheServiceWithSizeLimit(
+ final Configuration blobClientConfig,
+ final BlobView blobView,
+ @Nullable final InetSocketAddress serverAddress,
+ final long sizeListOfBlobCacheSizeTracker)
Review comment:
Do you mean `sizeLimitOfBlobCacheSizeTracker `?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
Review comment:
This is not a size limit but a blob count limit, maybe `maxAcceptedBlobCount`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0be481b90be72b85b8a294d9e14d08cad49ee40d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780)
* 911cd22a5de05ab8603c7f75fc58b22b96eef2fa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880494243
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 51db50340f762127f3c3c83dc86f0ac08f0877d9 (Thu Jul 15 08:15:24 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-23354).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-888721980
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c22129c7d12014419bc3a8d66934f97fb1df6a53 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914)
* 097a051078c80ca3f757f8734d484f84f08e7b21 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7095135b7315584613d01410819d8e3b78446dac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* b9d70021d437374b456556dce817d203bd4f21e2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070)
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 4175d2bca779b9a7f4dfeb9e9f5905777faea0e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096)
* ff39fb582bd29f5151b2a1625a3cf651c564d1f3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677233024
##########
File path: flink-core/src/main/java/org/apache/flink/configuration/BlobServerOptions.java
##########
@@ -118,4 +118,10 @@
key("blob.client.connect.timeout")
.defaultValue(0)
.withDescription("The connection timeout in milliseconds for the blob client.");
+
+ /** The maximum size limit of blob cache on TaskExecutor. */
+ public static final ConfigOption<Long> BLOB_CACHE_SIZE_LIMIT =
+ key("blob.cache.size.limit")
+ .defaultValue(1_024 * 1_024 * 1_024 * 10L)
+ .withDescription("The maximum size limit of blob cache for each TaskExecutor.");
Review comment:
This option has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r671088677
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/AbstractBlobCache.java
##########
@@ -249,6 +278,28 @@ public void close() throws IOException {
}
}
+ /**
+ * Delete the blob file with the given key.
+ *
+ * @param jobId ID of the job this blob belongs to (or <tt>null</tt> if job-unrelated)
+ * @param blobKey The key of the desired BLOB.
+ * @return file successfully deleted or not.
+ */
+ public boolean deleteFileUnsafely(@Nullable JobID jobId, BlobKey blobKey) {
+ final File localFile =
+ new File(
+ BlobUtils.getStorageLocationPath(
+ storageDir.getAbsolutePath(), jobId, blobKey));
+ if (!localFile.delete() && localFile.exists()) {
+ log.warn(
+ "Failed to delete locally cached BLOB {} at {}",
+ blobKey,
+ localFile.getAbsolutePath());
+ return false;
Review comment:
Why do we have a return value but add no handling for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* b9d70021d437374b456556dce817d203bd4f21e2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070)
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 4175d2bca779b9a7f4dfeb9e9f5905777faea0e3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* ec9fa78a8da0cbf99cb1a9a342275923176d3484 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031)
* 7ce8a08f3d996eee8c4ac7ac2de4618152099ac7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677945612
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -247,6 +404,7 @@ public void run() {
boolean success = false;
try {
+ blobCacheSizeTracker.unregisterJob(jobId);
Review comment:
Thanks for the explanation. I think you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #16498: [WIP][FLINK-23354][blob] Limit the size of blob cache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 51db50340f762127f3c3c83dc86f0ac08f0877d9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678029356
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
Review comment:
`BLOB_SIZE` does not make sense here. `data[i]` will the reassigned later.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
Review comment:
`last` -> `least`
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
+ readFileAndVerifyContent(cache, jobId[1], key[1], data[1]);
+
+ // Then get the fourth blob, make sure the third blob is replaced.
+ // Here we cannot get the first blob again, because for transient blob,
+ // BlobServer will delete the blob once it's transmitted.
+ key[3] = put(server, jobId[3], data[3], blobType);
+ readFileAndVerifyContent(cache, jobId[3], key[3], data[3]);
+ file[3] = getFile(cache, jobId[3], key[3]);
+
+ assertTrue(file[1].exists());
+ assertTrue(file[3].exists());
+ assertFalse(file[2].exists());
+ }
+ }
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcessConcurrently() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ final int concurrency = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(concurrency);
Review comment:
This kind of executor needs to be properly shutdown after used.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
+ readFileAndVerifyContent(cache, jobId[1], key[1], data[1]);
+
+ // Then get the fourth blob, make sure the third blob is replaced.
+ // Here we cannot get the first blob again, because for transient blob,
+ // BlobServer will delete the blob once it's transmitted.
+ key[3] = put(server, jobId[3], data[3], blobType);
+ readFileAndVerifyContent(cache, jobId[3], key[3], data[3]);
+ file[3] = getFile(cache, jobId[3], key[3]);
+
+ assertTrue(file[1].exists());
+ assertTrue(file[3].exists());
+ assertFalse(file[2].exists());
+ }
+ }
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcessConcurrently() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ final int concurrency = 3;
+ final ExecutorService executor = Executors.newFixedThreadPool(concurrency);
+
+ // The size limit is the size of 2 blobs
+ final int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[concurrency];
+ BlobKey[] key = new BlobKey[concurrency];
+ File[] file = new File[concurrency];
+
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ // Put all the BLOBs into the blob server one by one
+ for (int i = 0; i < concurrency; i++) {
+ jobId[i] = new JobID();
+
+ key[i] = put(server, jobId[i], blobData, blobType);
+ assertNotNull(key[i]);
Review comment:
all `blobData` are the same. Why are we not doing the same as in `testTrackSizeLimitAndDeleteExcess()`?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
Review comment:
Maybe add a comments to say that this is to make contents different from each other for later verification?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
Review comment:
Let's create a variable to host this number 4.
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
Review comment:
Can we extract the blob initialization into a method to be reused by all tests?
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are going to be inserted and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final int BLOB_SIZE = 10_000;
+
+ private static final Random RANDOM = new Random();
+
+ @Rule public final TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ @Test
+ public void testTrackSizeLimitAndDeleteExcess() throws Exception {
+
+ final BlobKey.BlobType blobType = BlobKey.BlobType.PERMANENT_BLOB;
+
+ final Configuration config = new Configuration();
+ config.setString(
+ BlobServerOptions.STORAGE_DIRECTORY, temporaryFolder.newFolder().getAbsolutePath());
+
+ // The size limit is the size of 2 blobs
+ int sizeLimit = 2;
+
+ try (BlobServer server = new BlobServer(config, new VoidBlobStore());
+ BlobCacheService cache =
+ initBlobCacheServiceWithSizeLimit(
+ config,
+ new VoidBlobStore(),
+ new InetSocketAddress("localhost", server.getPort()),
+ sizeLimit * BLOB_SIZE)) {
+
+ server.start();
+
+ // Initialize the data structures
+ JobID[] jobId = new JobID[4];
+ byte[][] data = new byte[4][BLOB_SIZE];
+ BlobKey[] key = new BlobKey[4];
+ File[] file = new File[4];
+
+ // Prepare the blob data
+ byte[] blobData = new byte[BLOB_SIZE];
+ RANDOM.nextBytes(blobData);
+
+ for (int i = 0; i < 4; i++) {
+ jobId[i] = new JobID();
+ data[i] = Arrays.copyOf(blobData, blobData.length);
+ data[i][i] ^= 1;
+ }
+
+ // Put the first, the second and the the third BLOB into the blob server one by one
+ // and let the cache retrieve the data from the blob
+ for (int i = 0; i < 3; i++) {
+ key[i] = put(server, jobId[i], data[i], blobType);
+ assertNotNull(key[i]);
+
+ readFileAndVerifyContent(cache, jobId[i], key[i], data[i]);
+ file[i] = getFile(cache, jobId[i], key[i]);
+ assertTrue(file[i].exists());
+ }
+
+ // Since the size limit of the cache is the size of 2 blobs,
+ // the first blob is removed and the second one remains
+ assertFalse(file[0].exists());
+ assertTrue(file[1].exists());
+
+ // Get the second blob once again, make the third blob to be the last recently used
+ readFileAndVerifyContent(cache, jobId[1], key[1], data[1]);
+
+ // Then get the fourth blob, make sure the third blob is replaced.
+ // Here we cannot get the first blob again, because for transient blob,
Review comment:
The blobs are not transient here. IIUC, this file deletion is not related to BlobServer, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #16498:
URL: https://github.com/apache/flink/pull/16498#issuecomment-880512519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20479",
"triggerID" : "51db50340f762127f3c3c83dc86f0ac08f0877d9",
"triggerType" : "PUSH"
}, {
"hash" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20780",
"triggerID" : "0be481b90be72b85b8a294d9e14d08cad49ee40d",
"triggerType" : "PUSH"
}, {
"hash" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20798",
"triggerID" : "911cd22a5de05ab8603c7f75fc58b22b96eef2fa",
"triggerType" : "PUSH"
}, {
"hash" : "7095135b7315584613d01410819d8e3b78446dac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20836",
"triggerID" : "7095135b7315584613d01410819d8e3b78446dac",
"triggerType" : "PUSH"
}, {
"hash" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20914",
"triggerID" : "c22129c7d12014419bc3a8d66934f97fb1df6a53",
"triggerType" : "PUSH"
}, {
"hash" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20969",
"triggerID" : "097a051078c80ca3f757f8734d484f84f08e7b21",
"triggerType" : "PUSH"
}, {
"hash" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea075fb62bc7853bcdbbf6d275c70048a9915f57",
"triggerType" : "PUSH"
}, {
"hash" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21004",
"triggerID" : "00805fc22fb911bbebe4efb0b5ffc4a29aeac25c",
"triggerType" : "PUSH"
}, {
"hash" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21031",
"triggerID" : "ec9fa78a8da0cbf99cb1a9a342275923176d3484",
"triggerType" : "PUSH"
}, {
"hash" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21038",
"triggerID" : "7ce8a08f3d996eee8c4ac7ac2de4618152099ac7",
"triggerType" : "PUSH"
}, {
"hash" : "b9d70021d437374b456556dce817d203bd4f21e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21070",
"triggerID" : "b9d70021d437374b456556dce817d203bd4f21e2",
"triggerType" : "PUSH"
}, {
"hash" : "c80d84283688488ccced44ab20354ba3185307df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c80d84283688488ccced44ab20354ba3185307df",
"triggerType" : "PUSH"
}, {
"hash" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096",
"triggerID" : "4175d2bca779b9a7f4dfeb9e9f5905777faea0e3",
"triggerType" : "PUSH"
}, {
"hash" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112",
"triggerID" : "ff39fb582bd29f5151b2a1625a3cf651c564d1f3",
"triggerType" : "PUSH"
}, {
"hash" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01f718e46e76efe7f74fa927a82a5329f5ef012b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea075fb62bc7853bcdbbf6d275c70048a9915f57 UNKNOWN
* c80d84283688488ccced44ab20354ba3185307df UNKNOWN
* 4175d2bca779b9a7f4dfeb9e9f5905777faea0e3 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21096)
* ff39fb582bd29f5151b2a1625a3cf651c564d1f3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21112)
* 01f718e46e76efe7f74fa927a82a5329f5ef012b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Thesharing commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
Thesharing commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678454272
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are intended to be stored and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final Random RANDOM = new Random();
+
+ private static final BlobKey.BlobType BLOB_TYPE = BlobKey.BlobType.PERMANENT_BLOB;
+ private static final int BLOB_SIZE = 10_000;
+ // The size limit is the size of 2 blobs
+ private static final int MAX_NUM_OF_ACCEPTED_BLOBS = 2;
+ private static final int TOTAL_NUM_OF_BLOBS = 3;
+
+ private static final Configuration CONFIG = new Configuration();
Review comment:
Yes, although the possibility of having the same JobID and BlobKey is low, but it still makes this case unstable. Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r677293483
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/blob/PermanentBlobCache.java
##########
@@ -90,6 +96,21 @@ public PermanentBlobCache(
@Nullable final InetSocketAddress serverAddress)
throws IOException {
+ this(
+ blobClientConfig,
+ blobView,
+ serverAddress,
+ new BlobCacheSizeTracker(MemorySize.ofMebiBytes(100).getBytes()));
Review comment:
I prefer to introduce a private const `DEFAULT_SIZE_LIMIT_MB` in `PermanentBlobCache` which is 100.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #16498: [FLINK-23354][blob] Limit the size of ShuffleDescriptors in PermanentBlobCache on TaskExecutor
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #16498:
URL: https://github.com/apache/flink/pull/16498#discussion_r678406858
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/blob/PermanentBlobCacheSizeLimitTest.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.flink.runtime.blob;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.BlobServerOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import static org.apache.flink.runtime.blob.BlobServerPutTest.put;
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test for using {@link BlobCacheSizeTracker} to track the size of BLOBs in {@link
+ * PermanentBlobCache}. When new BLOBs are intended to be stored and the size limit exceeds, {@link
+ * BlobCacheSizeTracker} will provide excess BLOBs for {@link PermanentBlobCache} to delete.
+ */
+public class PermanentBlobCacheSizeLimitTest {
+
+ private static final Random RANDOM = new Random();
+
+ private static final BlobKey.BlobType BLOB_TYPE = BlobKey.BlobType.PERMANENT_BLOB;
+ private static final int BLOB_SIZE = 10_000;
+ // The size limit is the size of 2 blobs
+ private static final int MAX_NUM_OF_ACCEPTED_BLOBS = 2;
+ private static final int TOTAL_NUM_OF_BLOBS = 3;
+
+ private static final Configuration CONFIG = new Configuration();
Review comment:
I think it's better not to make `CONFIG` and `TEMPORARY_FOLDER` static and shared between tests.
It's safer to initialize them before each test so that tests will not affect each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org