You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by di...@apache.org on 2026/09/16 17:50:59 UTC
(polaris) branch main updated: Bound async file-cleanup deletions with a configurable timeout (#5398)
This is an automated email from the ASF dual-hosted git repository.
dimas-b pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/polaris.git
The following commit(s) were added to refs/heads/main by this push:
new 7738e9bd41 Bound async file-cleanup deletions with a configurable timeout (#5398)
7738e9bd41 is described below
commit 7738e9bd41778d6e4f2e0ea54f497049b5cbafee
Author: Prithvi S <pr...@gmail.com>
AuthorDate: Wed Sep 16 23:20:55 2026 +0530
Bound async file-cleanup deletions with a configurable timeout (#5398)
* Bound async file-cleanup deletions with a configurable timeout
The manifest and batch file-cleanup handlers waited on their asynchronous
object-store deletions with an unbounded `Future.get()`/`join()`. If a
single delete stalled (a hung TLS socket, an unresponsive storage endpoint,
a black-holed connection), the wait never returned and pinned the
task-executor thread. Enough stalled purges could starve all cleanup while
storage kept growing, with no error surfaced.
Add `polaris.tasks.file-deletion-timeout` (default 1h) and a shared
`awaitCompletion` helper in the base handler that waits with that deadline.
On timeout the wait is abandoned and the task fails, so it follows the
normal retry path instead of blocking the thread. A zero or negative
timeout preserves the previous unbounded behavior. In-flight deletes are
idempotent, so a later retry is safe.
* Address review: config docs, properties example, honest timeout comment
- Regenerate the polaris.tasks config reference table to include the new
file-deletion-timeout option, and add a commented example alongside the
other polaris.tasks.* settings in application.properties.
- Drop the misleading isCancelled assertion and the cancel() call on
timeout: CompletableFuture.cancel ignores mayInterruptIfRunning and does
not propagate upstream, so the in-flight deletes keep running. Only the
wait is abandoned; the comment now says so explicitly.
* Make file-cleanup deletion timeout terminal instead of retrying
* Clarify that terminal timeout leaves the task persisted, not actively recovered
---
CHANGELOG.md | 1 +
.../src/main/resources/application.properties | 1 +
.../service/task/BatchFileCleanupTaskHandler.java | 15 +++--
.../service/task/FileCleanupTaskHandler.java | 43 +++++++++++++-
.../service/task/FileDeletionTimeoutException.java | 38 ++++++++++++
.../task/ManifestFileCleanupTaskHandler.java | 46 ++++++---------
.../polaris/service/task/TaskExecutorImpl.java | 40 +++++++++++--
.../service/task/TaskHandlerConfiguration.java | 10 ++++
.../task/BatchFileCleanupTaskHandlerTest.java | 5 +-
.../service/task/FileCleanupTaskHandlerTest.java | 67 ++++++++++++++++++++++
.../task/ManifestFileCleanupTaskHandlerTest.java | 3 +-
.../polaris/service/task/TaskExecutorImplTest.java | 58 +++++++++++++++++--
.../config-sections/smallrye-polaris_tasks.md | 1 +
13 files changed, 278 insertions(+), 50 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8d456244cc..7d7b76e392 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -124,6 +124,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
valid Base64 but not a serialized page token (garbage, truncated, or produced by an incompatible
Polaris version) previously escaped as Jackson decoding exceptions.
- Iceberg REST: renaming a table or view with a missing `source` or `destination` now returns `400 Bad Request` instead of `500 Internal Server Error`.
+- Async file-cleanup tasks now bound how long they wait for object-store deletions via the new `polaris.tasks.file-deletion-timeout` (default 1h), so a stalled storage endpoint can no longer pin a task-executor thread indefinitely; a timeout is terminal for the current run rather than immediately retried, so it does not stack more deletions onto the stalled endpoint.
- Python CLI `catalogs create --type external` now validates `--storage-type` and `--default-base-location` up front, matching the behavior for internal catalogs and the flags' documented "(Required)" status. Previously, omitting either produced an opaque pydantic `ValidationError` at request-build time.
- Iceberg REST: server-side JSON processing failures (HTTP 500) now return the standard Iceberg
error envelope (`{"error": {...}}`) instead of a flat `{"code", "message"}` body, so Iceberg
diff --git a/runtime/defaults/src/main/resources/application.properties b/runtime/defaults/src/main/resources/application.properties
index 6d55b929e0..2979e52f7f 100644
--- a/runtime/defaults/src/main/resources/application.properties
+++ b/runtime/defaults/src/main/resources/application.properties
@@ -222,6 +222,7 @@ polaris.metrics.tags.application=Polaris
# polaris.tasks.max-concurrent-tasks=100
# polaris.tasks.max-queued-tasks=1000
+# polaris.tasks.file-deletion-timeout=PT1H
polaris.rate-limiter.filter.type=no-op
polaris.rate-limiter.token-bucket.type=default
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandler.java b/runtime/service/src/main/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandler.java
index 8b4f0ecc84..c91890ecd7 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandler.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandler.java
@@ -18,6 +18,7 @@
*/
package org.apache.polaris.service.task;
+import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
@@ -42,8 +43,10 @@ public class BatchFileCleanupTaskHandler extends FileCleanupTaskHandler {
private static final Logger LOGGER = LoggerFactory.getLogger(BatchFileCleanupTaskHandler.class);
public BatchFileCleanupTaskHandler(
- TaskFileIOSupplier fileIOSupplier, ExecutorService executorService) {
- super(fileIOSupplier, executorService);
+ TaskFileIOSupplier fileIOSupplier,
+ ExecutorService executorService,
+ Duration fileDeletionTimeout) {
+ super(fileIOSupplier, executorService, fileDeletionTimeout);
}
@Override
@@ -85,12 +88,8 @@ public class BatchFileCleanupTaskHandler extends FileCleanupTaskHandler {
tryDelete(
tableId, authorizedFileIO, validFiles, cleanupTask.type().getValue(), true, null, 1);
- try {
- deleteFutures.join();
- } catch (Exception e) {
- LOGGER.error("Exception detected during batch files deletion", e);
- throw new RuntimeException(e);
- }
+ awaitCompletion(
+ deleteFutures, "batch deletion of " + validFiles.size() + " files for table " + tableId);
}
}
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/task/FileCleanupTaskHandler.java b/runtime/service/src/main/java/org/apache/polaris/service/task/FileCleanupTaskHandler.java
index 86c9c84519..d891267f72 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/task/FileCleanupTaskHandler.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/task/FileCleanupTaskHandler.java
@@ -18,9 +18,12 @@
*/
package org.apache.polaris.service.task;
+import java.time.Duration;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
import org.apache.iceberg.CatalogUtil;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.exceptions.NotFoundException;
@@ -45,12 +48,50 @@ public abstract class FileCleanupTaskHandler implements TaskHandler {
public static final int FILE_DELETION_RETRY_MILLIS = 100;
public final TaskFileIOSupplier fileIOSupplier;
public final ExecutorService executorService;
+ private final Duration fileDeletionTimeout;
private static final Logger LOGGER = LoggerFactory.getLogger(FileCleanupTaskHandler.class);
public FileCleanupTaskHandler(
- TaskFileIOSupplier fileIOSupplier, ExecutorService executorService) {
+ TaskFileIOSupplier fileIOSupplier,
+ ExecutorService executorService,
+ Duration fileDeletionTimeout) {
this.fileIOSupplier = fileIOSupplier;
this.executorService = executorService;
+ this.fileDeletionTimeout = fileDeletionTimeout;
+ }
+
+ /**
+ * Blocks until the given deletion stage completes, bounded by the configured {@link
+ * TaskHandlerConfiguration#fileDeletionTimeout()}. On timeout the wait is abandoned and a {@link
+ * FileDeletionTimeoutException} is thrown; this is a terminal failure that releases the
+ * task-executor thread without triggering an in-process retry against the still-stalled endpoint.
+ * A zero or negative timeout waits indefinitely, preserving the previous behavior.
+ *
+ * @param deletion the aggregate deletion stage to await
+ * @param description human-readable description of the work, used in the timeout message
+ */
+ protected void awaitCompletion(CompletableFuture<Void> deletion, String description) {
+ try {
+ if (fileDeletionTimeout.isZero() || fileDeletionTimeout.isNegative()) {
+ deletion.get();
+ } else {
+ deletion.get(fileDeletionTimeout.toMillis(), TimeUnit.MILLISECONDS);
+ }
+ } catch (TimeoutException e) {
+ // Only the wait is abandoned: the in-flight deletes keep running on the deletion executor
+ // and cannot be interrupted (CompletableFuture.cancel ignores mayInterruptIfRunning and does
+ // not propagate to the upstream runAsync tasks). Releasing this task-executor thread is the
+ // goal. A timeout is terminal for this execution rather than retried in-process, because an
+ // immediate retry would only stack more deletions onto the same stalled endpoint; the deletes
+ // are idempotent, so re-running the task later is safe.
+ throw new FileDeletionTimeoutException(
+ "Timed out after " + fileDeletionTimeout + " waiting for " + description, e);
+ } catch (ExecutionException e) {
+ throw new RuntimeException("Failed while waiting for " + description, e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for " + description, e);
+ }
}
@Override
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/task/FileDeletionTimeoutException.java b/runtime/service/src/main/java/org/apache/polaris/service/task/FileDeletionTimeoutException.java
new file mode 100644
index 0000000000..0bd20ee0dd
--- /dev/null
+++ b/runtime/service/src/main/java/org/apache/polaris/service/task/FileDeletionTimeoutException.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.task;
+
+/**
+ * Signals that a file-cleanup task exceeded its configured deletion timeout. This is treated as a
+ * terminal failure for the current execution: the task is not retried in-process, because an
+ * immediate retry would only pile a fresh generation of deletions onto an already-stalled endpoint
+ * (the previous deletions keep running and cannot be interrupted).
+ *
+ * <p>The task entity is left persisted rather than dropped. That is the same end state as an
+ * ordinary failure after in-process retries are exhausted, and it keeps the row eligible for
+ * lease-based recovery via {@code PolarisMetaStoreManager.loadTasks} (which re-leases tasks whose
+ * last attempt is older than {@code POLARIS_TASK_TIMEOUT_MILLIS}). Note that no runtime component
+ * currently drives that recovery, so in practice the persisted task is not picked back up again in
+ * either case; this only avoids the retry pile-up, it does not add a new recovery guarantee.
+ */
+class FileDeletionTimeoutException extends RuntimeException {
+ FileDeletionTimeoutException(String message, Throwable cause) {
+ super(message, cause);
+ }
+}
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandler.java b/runtime/service/src/main/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandler.java
index c3b4c78b5b..6be4848e4d 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandler.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandler.java
@@ -20,11 +20,11 @@ package org.apache.polaris.service.task;
import java.io.IOException;
import java.io.UncheckedIOException;
+import java.time.Duration;
import java.util.List;
import java.util.Spliterator;
import java.util.Spliterators;
import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.stream.StreamSupport;
import org.apache.iceberg.ContentFile;
@@ -51,8 +51,10 @@ public class ManifestFileCleanupTaskHandler extends FileCleanupTaskHandler {
LoggerFactory.getLogger(ManifestFileCleanupTaskHandler.class);
public ManifestFileCleanupTaskHandler(
- TaskFileIOSupplier fileIOSupplier, ExecutorService executorService) {
- super(fileIOSupplier, executorService);
+ TaskFileIOSupplier fileIOSupplier,
+ ExecutorService executorService,
+ Duration fileDeletionTimeout) {
+ super(fileIOSupplier, executorService, fileDeletionTimeout);
}
@Override
@@ -97,30 +99,20 @@ public class ManifestFileCleanupTaskHandler extends FileCleanupTaskHandler {
contentFileDeletes.size(),
manifestFile.path(),
manifestFile.content());
- try {
- // wait for all content files to be deleted, then wait for the manifest itself to be deleted
- CompletableFuture.allOf(contentFileDeletes.toArray(CompletableFuture[]::new))
- .thenCompose(
- (v) -> {
- LOGGER
- .atInfo()
- .addKeyValue(StructuredLogKeys.MANIFEST_FILE, manifestFile.path())
- .log("All content files in manifest deleted - deleting manifest");
- return tryDelete(
- tableId, fileIO, manifestFile.path(), manifestFile.path(), null, 1);
- })
- .get();
- return;
- } catch (InterruptedException e) {
- LOGGER.error(
- "Interrupted exception deleting content files from manifest {}",
- manifestFile.path(),
- e);
- throw new RuntimeException(e);
- } catch (ExecutionException e) {
- LOGGER.error("Unable to delete content files from manifest {}", manifestFile.path(), e);
- throw new RuntimeException(e);
- }
+ // wait for all content files to be deleted, then wait for the manifest itself to be deleted
+ CompletableFuture<Void> deletion =
+ CompletableFuture.allOf(contentFileDeletes.toArray(CompletableFuture[]::new))
+ .thenCompose(
+ (v) -> {
+ LOGGER
+ .atInfo()
+ .addKeyValue(StructuredLogKeys.MANIFEST_FILE, manifestFile.path())
+ .log("All content files in manifest deleted - deleting manifest");
+ return tryDelete(
+ tableId, fileIO, manifestFile.path(), manifestFile.path(), null, 1);
+ });
+ awaitCompletion(deletion, "deletion of files in manifest " + manifestFile.path());
+ return;
} catch (IOException e) {
// Catches from ManifestFiles.read() (resource creation), from inside the block
// (e.g. manifest iteration), or from close(). We throw (wrapping) so the original
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java b/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java
index c8d656c03b..cd9c5d4efc 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java
@@ -85,10 +85,11 @@ public class TaskExecutorImpl implements TaskExecutor {
private final PolarisEventDispatcher polarisEventDispatcher;
private final PolarisEventMetadataFactory eventMetadataFactory;
@Nullable private final Tracer tracer;
+ private final TaskHandlerConfiguration taskConfiguration;
@SuppressWarnings("unused") // Required by CDI
protected TaskExecutorImpl() {
- this(null, null, null, null, null, null, null, null, null, null, null);
+ this(null, null, null, null, null, null, null, null, null, null, null, null);
}
@Inject
@@ -104,7 +105,8 @@ public class TaskExecutorImpl implements TaskExecutor {
PolarisEventMetadataFactory eventMetadataFactory,
@Nullable Tracer tracer,
PolarisPrincipalHolder polarisPrincipalHolder,
- PolarisPrincipal polarisPrincipal) {
+ PolarisPrincipal polarisPrincipal,
+ TaskHandlerConfiguration taskConfiguration) {
this.executor = executor;
this.clock = clock;
this.metaStoreManagerFactory = metaStoreManagerFactory;
@@ -115,6 +117,7 @@ public class TaskExecutorImpl implements TaskExecutor {
this.tracer = tracer;
this.polarisPrincipalHolder = polarisPrincipalHolder;
this.polarisPrincipal = polarisPrincipal;
+ this.taskConfiguration = taskConfiguration;
if (errorHandler != null && errorHandler.isResolvable()) {
this.errorHandler = Optional.of(errorHandler.get());
@@ -129,10 +132,14 @@ public class TaskExecutorImpl implements TaskExecutor {
new TableCleanupTaskHandler(this, clock, metaStoreManagerFactory, fileIOSupplier));
addTaskHandler(
new ManifestFileCleanupTaskHandler(
- fileIOSupplier, Executors.newVirtualThreadPerTaskExecutor()));
+ fileIOSupplier,
+ Executors.newVirtualThreadPerTaskExecutor(),
+ taskConfiguration.fileDeletionTimeout()));
addTaskHandler(
new BatchFileCleanupTaskHandler(
- fileIOSupplier, Executors.newVirtualThreadPerTaskExecutor()));
+ fileIOSupplier,
+ Executors.newVirtualThreadPerTaskExecutor(),
+ taskConfiguration.fileDeletionTimeout()));
}
/**
@@ -191,14 +198,37 @@ public class TaskExecutorImpl implements TaskExecutor {
if (previousError != null) {
t.addSuppressed(previousError);
}
- LOGGER.warn("Failed to handle task entity id {}", taskEntityId, t);
errorHandler.ifPresent(h -> h.accept(taskEntityId, false, t));
+ if (!isRetryable(t)) {
+ LOGGER.warn(
+ "Task entity id {} failed with a terminal error; not retrying in-process. The"
+ + " task entity stays persisted (the same end state as exhausting retries).",
+ taskEntityId,
+ t);
+ return CompletableFuture.<Void>failedFuture(t);
+ }
+ LOGGER.warn("Failed to handle task entity id {}", taskEntityId, t);
return tryHandleTask(taskEntityId, callContext, eventMetadata, t, attempt + 1);
},
CompletableFuture.delayedExecutor(
TASK_RETRY_DELAY * (long) attempt, TimeUnit.MILLISECONDS, executor));
}
+ /**
+ * Determines whether a failed task attempt should be retried in-process. Most failures are
+ * transient and worth retrying, but a {@link FileDeletionTimeoutException} means an object-store
+ * deletion is stalled; retrying would only stack more (uncancellable) deletions onto the same
+ * endpoint, so such failures are treated as terminal for the current execution.
+ */
+ static boolean isRetryable(Throwable t) {
+ for (Throwable cause = t; cause != null; cause = cause.getCause()) {
+ if (cause instanceof FileDeletionTimeoutException) {
+ return false;
+ }
+ }
+ return true;
+ }
+
protected void handleTask(
long taskEntityId, CallContext ctx, PolarisEventMetadata eventMetadata, int attempt) {
if (polarisEventDispatcher.hasListeners(PolarisEventType.BEFORE_ATTEMPT_TASK)) {
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/task/TaskHandlerConfiguration.java b/runtime/service/src/main/java/org/apache/polaris/service/task/TaskHandlerConfiguration.java
index 76ef754cc8..8e433f26e0 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/task/TaskHandlerConfiguration.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/task/TaskHandlerConfiguration.java
@@ -20,6 +20,7 @@ package org.apache.polaris.service.task;
import io.smallrye.config.ConfigMapping;
import io.smallrye.config.WithDefault;
+import java.time.Duration;
@ConfigMapping(prefix = "polaris.tasks")
public interface TaskHandlerConfiguration {
@@ -29,4 +30,13 @@ public interface TaskHandlerConfiguration {
@WithDefault("-1")
int maxQueuedTasks();
+
+ /**
+ * Maximum time a file-cleanup task will wait for its asynchronous object-store deletions to
+ * complete before giving up and failing the task (which then follows the normal retry path). This
+ * bounds the task-executor thread so that a stalled storage endpoint cannot pin it indefinitely.
+ * A zero or negative value disables the deadline and waits indefinitely.
+ */
+ @WithDefault("PT1H")
+ Duration fileDeletionTimeout();
}
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandlerTest.java b/runtime/service/src/test/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandlerTest.java
index 4d3372719b..cb7fc49de3 100644
--- a/runtime/service/src/test/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandlerTest.java
+++ b/runtime/service/src/test/java/org/apache/polaris/service/task/BatchFileCleanupTaskHandlerTest.java
@@ -29,6 +29,7 @@ import io.quarkus.test.junit.QuarkusMock;
import io.quarkus.test.junit.QuarkusTest;
import jakarta.inject.Inject;
import java.io.IOException;
+import java.time.Duration;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
@@ -81,7 +82,7 @@ public class BatchFileCleanupTaskHandlerTest {
private BatchFileCleanupTaskHandler newBatchFileCleanupTaskHandler(FileIO fileIO) {
Mockito.when(taskFileIOSupplier.apply(Mockito.any(), Mockito.any())).thenReturn(fileIO);
- return new BatchFileCleanupTaskHandler(taskFileIOSupplier, executor);
+ return new BatchFileCleanupTaskHandler(taskFileIOSupplier, executor, Duration.ofMinutes(5));
}
@Test
@@ -286,7 +287,7 @@ public class BatchFileCleanupTaskHandlerTest {
TableIdentifier tableIdentifier = TableIdentifier.of(Namespace.of("db1", "schema1"), "table1");
Mockito.when(taskFileIOSupplier.apply(Mockito.any(), Mockito.any())).thenReturn(fileIO);
BatchFileCleanupTaskHandler handler =
- new BatchFileCleanupTaskHandler(taskFileIOSupplier, executor) {
+ new BatchFileCleanupTaskHandler(taskFileIOSupplier, executor, Duration.ofMinutes(5)) {
@Override
public CompletableFuture<Void> tryDelete(
TableIdentifier tableId,
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/task/FileCleanupTaskHandlerTest.java b/runtime/service/src/test/java/org/apache/polaris/service/task/FileCleanupTaskHandlerTest.java
new file mode 100644
index 0000000000..590640b011
--- /dev/null
+++ b/runtime/service/src/test/java/org/apache/polaris/service/task/FileCleanupTaskHandlerTest.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.task;
+
+import static org.assertj.core.api.Assertions.assertThatCode;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import java.time.Duration;
+import java.util.concurrent.CompletableFuture;
+import org.junit.jupiter.api.Test;
+
+/** Unit tests for the shared {@link FileCleanupTaskHandler#awaitCompletion} deadline logic. */
+class FileCleanupTaskHandlerTest {
+
+ private static FileCleanupTaskHandler handler(Duration timeout) {
+ // Only the timeout is exercised here; the file IO supplier and executor are unused.
+ return new BatchFileCleanupTaskHandler(null, null, timeout);
+ }
+
+ @Test
+ void awaitCompletionFailsWhenDeletionStalls() {
+ // A stalled deletion must free the waiting thread with a terminal timeout failure rather than
+ // block forever. Note this only abandons the wait; the in-flight deletes cannot be interrupted.
+ CompletableFuture<Void> neverCompletes = new CompletableFuture<>();
+ assertThatThrownBy(
+ () ->
+ handler(Duration.ofMillis(50)).awaitCompletion(neverCompletes, "stalled deletion"))
+ .isInstanceOf(FileDeletionTimeoutException.class)
+ .hasMessageContaining("Timed out")
+ .hasMessageContaining("stalled deletion");
+ }
+
+ @Test
+ void awaitCompletionReturnsWhenDeletionCompletes() {
+ assertThatCode(
+ () ->
+ handler(Duration.ofSeconds(30))
+ .awaitCompletion(CompletableFuture.completedFuture(null), "deletion"))
+ .doesNotThrowAnyException();
+ }
+
+ @Test
+ void awaitCompletionWithNonPositiveTimeoutDoesNotApplyDeadline() {
+ // A zero/negative timeout disables the deadline; a completed future must still return.
+ assertThatCode(
+ () ->
+ handler(Duration.ZERO)
+ .awaitCompletion(CompletableFuture.completedFuture(null), "deletion"))
+ .doesNotThrowAnyException();
+ }
+}
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandlerTest.java b/runtime/service/src/test/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandlerTest.java
index d3d4b68308..86db02a13c 100644
--- a/runtime/service/src/test/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandlerTest.java
+++ b/runtime/service/src/test/java/org/apache/polaris/service/task/ManifestFileCleanupTaskHandlerTest.java
@@ -27,6 +27,7 @@ import io.quarkus.test.junit.QuarkusMock;
import io.quarkus.test.junit.QuarkusTest;
import jakarta.inject.Inject;
import java.io.IOException;
+import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
@@ -73,7 +74,7 @@ class ManifestFileCleanupTaskHandlerTest {
private ManifestFileCleanupTaskHandler newManifestFileCleanupTaskHandler(FileIO fileIO) {
Mockito.when(taskFileIOSupplier.apply(Mockito.any(), Mockito.any())).thenReturn(fileIO);
- return new ManifestFileCleanupTaskHandler(taskFileIOSupplier, executor);
+ return new ManifestFileCleanupTaskHandler(taskFileIOSupplier, executor, Duration.ofMinutes(5));
}
@Test
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java b/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java
index d58955bc14..55f7a61846 100644
--- a/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java
+++ b/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java
@@ -22,7 +22,10 @@ import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import java.time.Duration;
import java.util.Map;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.polaris.core.PolarisCallContext;
import org.apache.polaris.core.config.RealmConfig;
@@ -52,6 +55,26 @@ import org.mockito.Mockito;
/** Unit tests for TaskExecutorImpl */
public class TaskExecutorImplTest {
+
+ private static TaskHandlerConfiguration taskHandlerConfiguration() {
+ return new TaskHandlerConfiguration() {
+ @Override
+ public int maxConcurrentTasks() {
+ return -1;
+ }
+
+ @Override
+ public int maxQueuedTasks() {
+ return -1;
+ }
+
+ @Override
+ public Duration fileDeletionTimeout() {
+ return Duration.ofHours(1);
+ }
+ };
+ }
+
@Test
void testEventsAreEmitted() {
String realm = "myrealm";
@@ -91,7 +114,8 @@ public class TaskExecutorImplTest {
testServices.eventMetadataFactory(),
null,
new PolarisPrincipalHolder(),
- testServices.principal());
+ testServices.principal(),
+ taskHandlerConfiguration());
executor.addTaskHandler(
new TaskHandler() {
@@ -162,7 +186,8 @@ public class TaskExecutorImplTest {
testServices.eventMetadataFactory(),
null,
new PolarisPrincipalHolder(),
- testServices.principal());
+ testServices.principal(),
+ taskHandlerConfiguration());
// No handlers registered
assertThatThrownBy(
@@ -211,7 +236,8 @@ public class TaskExecutorImplTest {
testServices.eventMetadataFactory(),
null,
new PolarisPrincipalHolder(),
- testServices.principal());
+ testServices.principal(),
+ taskHandlerConfiguration());
executor.addTaskHandler(
new TaskHandler() {
@@ -274,7 +300,8 @@ public class TaskExecutorImplTest {
testServices.eventMetadataFactory(),
null,
new PolarisPrincipalHolder(),
- testServices.principal());
+ testServices.principal(),
+ taskHandlerConfiguration());
executor.addTaskHandler(
new TaskHandler() {
@@ -382,7 +409,8 @@ public class TaskExecutorImplTest {
testServices.eventMetadataFactory(),
null,
new PolarisPrincipalHolder(),
- testServices.principal());
+ testServices.principal(),
+ taskHandlerConfiguration());
executor.addTaskHandler(
new TaskHandler() {
@@ -451,7 +479,8 @@ public class TaskExecutorImplTest {
testServices.eventMetadataFactory(),
null,
new PolarisPrincipalHolder(),
- testServices.principal());
+ testServices.principal(),
+ taskHandlerConfiguration());
executor.addTaskHandler(
new TaskHandler() {
@@ -478,4 +507,21 @@ public class TaskExecutorImplTest {
// We verify at least the first call happened (throw leads to exception path).
assertThat(handlerCalls.get()).isGreaterThanOrEqualTo(1);
}
+
+ @Test
+ void fileDeletionTimeoutIsTerminalAndNotRetried() {
+ // Ordinary failures are retried; a stalled-deletion timeout is terminal for the current
+ // execution so it is not retried in-process against the still-stalled endpoint.
+ assertThat(TaskExecutorImpl.isRetryable(new RuntimeException("transient"))).isTrue();
+ assertThat(
+ TaskExecutorImpl.isRetryable(
+ new FileDeletionTimeoutException("timed out", new TimeoutException())))
+ .isFalse();
+ // The timeout may be wrapped by the CompletableFuture machinery before reaching the retry path.
+ assertThat(
+ TaskExecutorImpl.isRetryable(
+ new CompletionException(
+ new FileDeletionTimeoutException("timed out", new TimeoutException()))))
+ .isFalse();
+ }
}
diff --git a/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_tasks.md b/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_tasks.md
index d895daa526..bac1a45083 100644
--- a/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_tasks.md
+++ b/site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_tasks.md
@@ -27,3 +27,4 @@ build:
|----------|---------------|------|-------------|
| `polaris.tasks.max-concurrent-tasks` | `-1` | `int` | |
| `polaris.tasks.max-queued-tasks` | `-1` | `int` | |
+| `polaris.tasks.file-deletion-timeout` | `PT1H` | `duration` | Maximum time a file-cleanup task will wait for its asynchronous object-store deletions to complete before giving up and failing the task (which then follows the normal retry path). This bounds the task-executor thread so that a stalled storage endpoint cannot pin it indefinitely. A zero or negative value disables the deadline and waits indefinitely. |