You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by di...@apache.org on 2026/09/14 19:19:58 UTC
(polaris) branch main updated: Handle an already-dropped task entity on async task retry (#5312)
This is an automated email from the ASF dual-hosted git repository.
dimas-b pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/polaris.git
The following commit(s) were added to refs/heads/main by this push:
new 30295510e9 Handle an already-dropped task entity on async task retry (#5312)
30295510e9 is described below
commit 30295510e9db73e1c0771ac375dde9f2fe42f9d5
Author: Sepuri Sai Krishna <sa...@gmail.com>
AuthorDate: Tue Sep 15 00:49:52 2026 +0530
Handle an already-dropped task entity on async task retry (#5312)
* Handle an already-dropped task entity on async task retry
TaskExecutorImpl.handleTask loads the task entity and immediately calls
getType() on it. PolarisMetaStoreManager.loadEntity returns an EntityResult
with a null entity when the id is not found, so the call throws a
NullPointerException whenever the task entity is gone by the time an attempt
runs.
A retry reaches exactly that state by design. As the comment above the
dropEntityIfExists call notes, a handler can complete successfully and the
subsequent entity drop can still fail, and that exception is rethrown so the
attempt is retried. The task entity may well have been dropped before the
failure, so the retry reloads nothing and dies on the null dereference. The
result is that a benign "work already done" case consumes all three attempts,
spaced by the retry backoff, and logs a NullPointerException each time instead
of a meaningful message.
A missing task entity now short-circuits the attempt: it is reported as a
successful, already-completed task and the attempt returns without invoking a
handler, so no further retries are scheduled.
* Distinguish a missing task entity from a failed load
Address review feedback: keying the "already completed" branch off
EntityResult#getEntity() == null treated every load failure as a
completed task, because EntityResult carries a null entity for all
non-success statuses. A retryable failure such as
TARGET_ENTITY_CONCURRENTLY_MODIFIED would have been recorded as
TASK_SUCCESS=true and the task abandoned.
Key the branch off getReturnStatus() instead: ENTITY_NOT_FOUND means
already completed, any other non-success status throws so the attempt
fails and is retried.
---
CHANGELOG.md | 1 +
.../polaris/service/task/TaskExecutorImpl.java | 33 +++-
.../polaris/service/task/TaskExecutorImplTest.java | 187 +++++++++++++++++++++
3 files changed, 217 insertions(+), 4 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c03463b73d..1d055a7cbe 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -181,6 +181,7 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
`SupportsBulkOperations` when the wrapped `FileIO` supports it, which affects every storage
backend, since `S3FileIO`, `GCSFileIO`, `ADLSFileIO` and `HadoopFileIO` all implement
`DelegateFileIO`.
+- Async task retries no longer fail with a `NullPointerException` when the task entity has already been dropped by a previous attempt. Such a retry is now recognized as an already-completed task and exits cleanly, instead of exhausting all retry attempts and logging a `NullPointerException` on each one.
### Commits
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java b/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java
index 88aaac64ce..c8d656c03b 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/task/TaskExecutorImpl.java
@@ -48,6 +48,8 @@ import org.apache.polaris.core.entity.PolarisEntityType;
import org.apache.polaris.core.entity.TaskEntity;
import org.apache.polaris.core.persistence.MetaStoreManagerFactory;
import org.apache.polaris.core.persistence.PolarisMetaStoreManager;
+import org.apache.polaris.core.persistence.dao.entity.BaseResult;
+import org.apache.polaris.core.persistence.dao.entity.EntityResult;
import org.apache.polaris.service.context.catalog.PolarisPrincipalHolder;
import org.apache.polaris.service.context.catalog.RealmContextHolder;
import org.apache.polaris.service.events.EventAttributes;
@@ -216,10 +218,33 @@ public class TaskExecutorImpl implements TaskExecutor {
LOGGER.info("Handling task entity id {}", taskEntityId);
PolarisMetaStoreManager metaStoreManager =
metaStoreManagerFactory.getOrCreateMetaStoreManager(ctx.getRealmContext());
- taskEntity =
- metaStoreManager
- .loadEntity(ctx.getPolarisCallContext(), 0L, taskEntityId, PolarisEntityType.TASK)
- .getEntity();
+ EntityResult loadResult =
+ metaStoreManager.loadEntity(
+ ctx.getPolarisCallContext(), 0L, taskEntityId, PolarisEntityType.TASK);
+ if (loadResult.getReturnStatus() == BaseResult.ReturnStatus.ENTITY_NOT_FOUND) {
+ // The task entity is gone, so an earlier attempt already handled it and dropped the
+ // entity. A retry reaches this point when the handler completed but the subsequent
+ // dropEntityIfExists failed and was rethrown (see below). There is no work left to do,
+ // so report the attempt as successful.
+ success = true;
+ LOGGER
+ .atInfo()
+ .addKeyValue(StructuredLogKeys.TASK_ENTITY_ID, taskEntityId)
+ .log("Task entity no longer exists; treating the task as already completed");
+ return;
+ }
+ if (!loadResult.isSuccess()) {
+ // Every other non-success status is a load failure, not evidence that the task is done.
+ // EntityResult carries a null entity for all of them, so the status is what distinguishes
+ // "already dropped" from "could not be read"; some, such as
+ // TARGET_ENTITY_CONCURRENTLY_MODIFIED, are explicitly retryable. Fail the attempt so it is
+ // retried instead of being recorded as a success and dropped.
+ throw new IllegalStateException(
+ String.format(
+ "Failed to load task entity id %d: %s",
+ taskEntityId, loadResult.getReturnStatus()));
+ }
+ taskEntity = loadResult.getEntity();
if (!PolarisEntityType.TASK.equals(taskEntity.getType())) {
throw new IllegalArgumentException("Provided taskId must be a task entity type");
}
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java b/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java
index 91369e647f..d58955bc14 100644
--- a/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java
+++ b/runtime/service/src/test/java/org/apache/polaris/service/task/TaskExecutorImplTest.java
@@ -19,15 +19,25 @@
package org.apache.polaris.service.task;
import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.polaris.core.PolarisCallContext;
+import org.apache.polaris.core.config.RealmConfig;
import org.apache.polaris.core.context.CallContext;
import org.apache.polaris.core.context.RealmContext;
import org.apache.polaris.core.entity.AsyncTaskType;
import org.apache.polaris.core.entity.TaskEntity;
+import org.apache.polaris.core.persistence.BasePersistence;
+import org.apache.polaris.core.persistence.MetaStoreManagerFactory;
import org.apache.polaris.core.persistence.PolarisMetaStoreManager;
+import org.apache.polaris.core.persistence.bootstrap.RootCredentialsSet;
+import org.apache.polaris.core.persistence.cache.EntityCache;
+import org.apache.polaris.core.persistence.dao.entity.BaseResult;
+import org.apache.polaris.core.persistence.dao.entity.EntityResult;
+import org.apache.polaris.core.persistence.dao.entity.PrincipalSecretsResult;
import org.apache.polaris.service.TestServices;
import org.apache.polaris.service.context.catalog.PolarisPrincipalHolder;
import org.apache.polaris.service.context.catalog.RealmContextHolder;
@@ -38,6 +48,7 @@ import org.apache.polaris.service.events.PolarisEventType;
import org.apache.polaris.service.events.listeners.InMemoryEventCollector;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
/** Unit tests for TaskExecutorImpl */
public class TaskExecutorImplTest {
@@ -231,6 +242,182 @@ public class TaskExecutorImplTest {
assertThat(afterEvent.attributes().getRequired(EventAttributes.TASK_SUCCESS)).isEqualTo(false);
}
+ @Test
+ void handleTaskIsANoOpWhenTaskEntityNoLongerExists() {
+ String realm = "myrealm";
+ RealmContext realmContext = () -> realm;
+
+ TestServices testServices = TestServices.builder().realmContext(realmContext).build();
+
+ InMemoryEventCollector testPolarisEventDispatcher =
+ (InMemoryEventCollector) testServices.polarisEventDispatcher();
+
+ PolarisMetaStoreManager metaStoreManager = testServices.metaStoreManager();
+ PolarisCallContext polarisCallCtx = testServices.newCallContext();
+
+ // Reserve an id but never persist the entity, reproducing a retry that runs after an earlier
+ // attempt already handled the task and dropped its entity.
+ long droppedTaskId = metaStoreManager.generateNewEntityId(polarisCallCtx).getId();
+
+ AtomicInteger handlerCalls = new AtomicInteger(0);
+
+ TaskExecutorImpl executor =
+ new TaskExecutorImpl(
+ Runnable::run,
+ null,
+ testServices.clock(),
+ testServices.metaStoreManagerFactory(),
+ new TaskFileIOSupplier(
+ testServices.fileIOFactory(), testServices.storageAccessConfigProvider()),
+ new RealmContextHolder(),
+ testServices.polarisEventDispatcher(),
+ testServices.eventMetadataFactory(),
+ null,
+ new PolarisPrincipalHolder(),
+ testServices.principal());
+
+ executor.addTaskHandler(
+ new TaskHandler() {
+ @Override
+ public boolean canHandleTask(TaskEntity task) {
+ return true;
+ }
+
+ @Override
+ public void handleTask(TaskEntity task, CallContext callContext) {
+ handlerCalls.incrementAndGet();
+ }
+ });
+
+ assertThatCode(
+ () ->
+ executor.handleTask(
+ droppedTaskId,
+ polarisCallCtx,
+ PolarisEventMetadata.builder().realmId(realm).build(),
+ 2))
+ .doesNotThrowAnyException();
+
+ assertThat(handlerCalls.get()).isZero();
+
+ PolarisEvent afterEvent =
+ testPolarisEventDispatcher.getLatest(PolarisEventType.AFTER_ATTEMPT_TASK);
+ assertThat(afterEvent.attributes().getRequired(EventAttributes.TASK_ENTITY_ID))
+ .isEqualTo(droppedTaskId);
+ assertThat(afterEvent.attributes().getRequired(EventAttributes.TASK_SUCCESS)).isEqualTo(true);
+ }
+
+ @Test
+ void handleTaskFailsTheAttemptWhenLoadingTheTaskEntityFails() {
+ String realm = "myrealm";
+ RealmContext realmContext = () -> realm;
+
+ TestServices testServices = TestServices.builder().realmContext(realmContext).build();
+
+ InMemoryEventCollector testPolarisEventDispatcher =
+ (InMemoryEventCollector) testServices.polarisEventDispatcher();
+
+ PolarisMetaStoreManager metaStoreManager = testServices.metaStoreManager();
+ PolarisCallContext polarisCallCtx = testServices.newCallContext();
+
+ TaskEntity taskEntity =
+ new TaskEntity.Builder()
+ .setName("unreadable-task")
+ .setId(metaStoreManager.generateNewEntityId(polarisCallCtx).getId())
+ .setCreateTimestamp(testServices.clock().millis())
+ .build();
+ metaStoreManager.createEntityIfNotExists(polarisCallCtx, null, taskEntity);
+
+ // The entity exists; the load fails for a reason other than "not found". EntityResult reports
+ // every non-success status with a null entity, so only the status separates this case from an
+ // entity an earlier attempt already dropped.
+ PolarisMetaStoreManager failingMetaStoreManager = Mockito.spy(metaStoreManager);
+ Mockito.doReturn(new EntityResult(BaseResult.ReturnStatus.UNEXPECTED_ERROR_SIGNALED, null))
+ .when(failingMetaStoreManager)
+ .loadEntity(Mockito.any(), Mockito.anyLong(), Mockito.anyLong(), Mockito.any());
+
+ MetaStoreManagerFactory delegate = testServices.metaStoreManagerFactory();
+ MetaStoreManagerFactory failingFactory =
+ new MetaStoreManagerFactory() {
+ @Override
+ public PolarisMetaStoreManager getOrCreateMetaStoreManager(RealmContext realmContext) {
+ return failingMetaStoreManager;
+ }
+
+ @Override
+ public BasePersistence getOrCreateSession(RealmContext realmContext) {
+ return delegate.getOrCreateSession(realmContext);
+ }
+
+ @Override
+ public EntityCache getOrCreateEntityCache(
+ RealmContext realmContext, RealmConfig realmConfig) {
+ return delegate.getOrCreateEntityCache(realmContext, realmConfig);
+ }
+
+ @Override
+ public Map<String, PrincipalSecretsResult> bootstrapRealms(
+ Iterable<String> realms, RootCredentialsSet rootCredentialsSet) {
+ return delegate.bootstrapRealms(realms, rootCredentialsSet);
+ }
+
+ @Override
+ public Map<String, BaseResult> purgeRealms(Iterable<String> realms) {
+ return delegate.purgeRealms(realms);
+ }
+ };
+
+ AtomicInteger handlerCalls = new AtomicInteger(0);
+
+ TaskExecutorImpl executor =
+ new TaskExecutorImpl(
+ Runnable::run,
+ null,
+ testServices.clock(),
+ failingFactory,
+ new TaskFileIOSupplier(
+ testServices.fileIOFactory(), testServices.storageAccessConfigProvider()),
+ new RealmContextHolder(),
+ testServices.polarisEventDispatcher(),
+ testServices.eventMetadataFactory(),
+ null,
+ new PolarisPrincipalHolder(),
+ testServices.principal());
+
+ executor.addTaskHandler(
+ new TaskHandler() {
+ @Override
+ public boolean canHandleTask(TaskEntity task) {
+ return true;
+ }
+
+ @Override
+ public void handleTask(TaskEntity task, CallContext callContext) {
+ handlerCalls.incrementAndGet();
+ }
+ });
+
+ assertThatThrownBy(
+ () ->
+ executor.handleTask(
+ taskEntity.getId(),
+ polarisCallCtx,
+ PolarisEventMetadata.builder().realmId(realm).build(),
+ 2))
+ .isInstanceOf(IllegalStateException.class)
+ .hasMessageContaining("UNEXPECTED_ERROR_SIGNALED");
+
+ assertThat(handlerCalls.get()).isZero();
+
+ // The attempt must be reported as a failure so the retry logic runs again, rather than as a
+ // success that silently abandons a task whose entity is still present.
+ PolarisEvent afterEvent =
+ testPolarisEventDispatcher.getLatest(PolarisEventType.AFTER_ATTEMPT_TASK);
+ assertThat(afterEvent.attributes().getRequired(EventAttributes.TASK_ENTITY_ID))
+ .isEqualTo(taskEntity.getId());
+ assertThat(afterEvent.attributes().getRequired(EventAttributes.TASK_SUCCESS)).isEqualTo(false);
+ }
+
@Test
void asyncRetryIsTriggeredWhenHandlerThrows() throws InterruptedException {
String realm = "myrealm";