You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by li...@apache.org on 2026/08/22 10:05:39 UTC

(seatunnel) branch davidzollo_republish_11727_start_latch_release created (now e6d5214c3c)

This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a change to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git


      at e6d5214c3c [Fix][Zeta] Rename startSignalled to startLatchReleased

This branch includes the following new commits:

     new 4cbe31d164 [Fix][Zeta] Release the start latch when a blocking worker cannot start
     new 25a5766ea3 [Test][Zeta] Add a regression test for the deploy hang in #11679
     new e8e1eb7a72 [Fix][Zeta] Guard recycleClassLoader against a missing execution context
     new fc758e3054 [Test][Zeta] Clarify the review points on the regression test and Javadoc
     new 4179d0da19 [Fix][Zeta] Guard the finished-context handoff in taskDone against the same race
     new 85bdfc189f [Test][Zeta] Assert the task group future settles on the stale-context path
     new e6d5214c3c [Fix][Zeta] Rename startSignalled to startLatchReleased

The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



(seatunnel) 05/07: [Fix][Zeta] Guard the finished-context handoff in taskDone against the same race

Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit 4179d0da19e0d7d044a2ad11dc27596538ce5822
Author: SAADAOUI ABDESSALEM <10...@users.noreply.github.com>
AuthorDate: Mon Aug 10 13:22:27 2026 +0200

    [Fix][Zeta] Guard the finished-context handoff in taskDone against the same race
    
    executionContexts.remove() returns null when a stale taskDone() from an earlier
    restore generation has already removed the entry - the same window that motivated
    the recycleClassLoader guard one line above. finishedExecutionContexts is a
    ConcurrentHashMap, which rejects null values, so the unguarded put threw out of
    BlockingWorker.run()'s finally block before future.complete() was reached.
    
    That left the task group's TaskExecutionState future - the one PhysicalVertex and
    SubPlan wait on - pending forever, reproducing a hang of the same class as #11679
    one level up. Skip the handoff when there is nothing to hand off.
    
    Raised as Issue 1 in review.
    
    (cherry picked from commit dba310df19b6379b8464da49bda20fadb3567ab8)
---
 .../apache/seatunnel/engine/server/TaskExecutionService.java | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
index f89f18c1ba..0a278f21f3 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
@@ -1421,8 +1421,16 @@ public class TaskExecutionService implements DynamicMetricsProvider {
             Throwable ex = executionException.get();
             if (completionLatch.decrementAndGet() == 0) {
                 recycleClassLoader(taskGroupLocation);
-                finishedExecutionContexts.put(
-                        taskGroupLocation, executionContexts.remove(taskGroupLocation));
+                // Same race as the guard in recycleClassLoader above: a stale taskDone()
+                // from an earlier restore generation may already have removed this entry,
+                // in which case remove() returns null. finishedExecutionContexts is a
+                // ConcurrentHashMap, which rejects null values, so an unguarded put would
+                // throw out of this finally block and skip future.complete() below -
+                // leaving the task group's completion future pending forever.
+                TaskGroupContext finishedContext = executionContexts.remove(taskGroupLocation);
+                if (finishedContext != null) {
+                    finishedExecutionContexts.put(taskGroupLocation, finishedContext);
+                }
                 cancellationFutures.remove(taskGroupLocation);
                 try {
                     cancelAsyncFunction(taskGroupLocation);


(seatunnel) 02/07: [Test][Zeta] Add a regression test for the deploy hang in #11679

Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit 25a5766ea3f22a298fefab5e8e778a1390085628
Author: SAADAOUI ABDESSALEM <10...@users.noreply.github.com>
AuthorDate: Sun Aug 9 16:18:02 2026 +0200

    [Test][Zeta] Add a regression test for the deploy hang in #11679
    
    Asserts the narrow contract that prevents the hang: deploying a task group must
    return even if its execution context disappears while the deployment is in
    flight. On the unpatched engine it fails after 30s on the first iteration; with
    the fix the same 30 iterations complete in under 4s.
    
    (cherry picked from commit 23011fd7d4812a9e28df83fe4ce2430645d77e6e)
---
 .../server/TaskDeployStaleContextRaceTest.java     | 166 +++++++++++++++++++++
 1 file changed, 166 insertions(+)

diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
new file mode 100644
index 0000000000..b3b7357fc4
--- /dev/null
+++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.engine.server;
+
+import org.apache.seatunnel.engine.server.execution.TaskDeployState;
+import org.apache.seatunnel.engine.server.execution.TaskGroupContext;
+import org.apache.seatunnel.engine.server.execution.TaskGroupLocation;
+import org.apache.seatunnel.engine.server.execution.TaskGroupType;
+import org.apache.seatunnel.engine.server.execution.TestTask;
+import org.apache.seatunnel.engine.server.task.TaskGroupImmutableInformation;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import com.hazelcast.internal.serialization.Data;
+
+import java.lang.reflect.Field;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static java.util.Collections.emptySet;
+
+/**
+ * Regression test for <a href="https://github.com/apache/seatunnel/issues/11679">#11679</a>.
+ *
+ * <p>{@code TaskGroupLocation} is {jobId, pipelineId, taskGroupId} and is reused verbatim across
+ * pipeline restore generations. {@code TaskGroupExecutionTracker.taskDone()} removes the entry for
+ * that location from {@code executionContexts}, so a late {@code taskDone()} belonging to the
+ * previous generation can delete the context that the current generation's {@code
+ * deployLocalTask()} has just installed.
+ *
+ * <p>When that happens, {@code BlockingWorker.run()} resolves its class loader through {@code
+ * executionContexts.get(location)} <em>before</em> its {@code try} block, and {@code
+ * startedLatch.countDown()} sits <em>inside</em> it. A missing context therefore throws before the
+ * latch is ever counted down, the exception is swallowed by the submitting {@code Future}, and
+ * {@code submitBlockingTask()} waits on {@code startedLatch.await()} forever — while holding the
+ * {@code SubPlan} monitor, which in turn blocks checkpoint-error handling from ever moving the
+ * pipeline to a terminal state.
+ *
+ * <p>This test asserts the narrow contract that prevents the hang: <b>deploying a task group must
+ * return, even if the execution context for that location disappears while the deployment is in
+ * flight.</b> Whether it returns successfully or throws is not asserted — only that it does not
+ * block indefinitely.
+ *
+ * <p>The race window is small, so the test repeats the deployment and races a remover thread
+ * against it. It can therefore fail to <em>detect</em> a regression on an unlucky run, but it
+ * cannot report a failure that is not real: the only way it fails is a deployment that never
+ * returns.
+ */
+public class TaskDeployStaleContextRaceTest extends AbstractSeaTunnelServerTest {
+
+    /** Deployment is a local, in-memory operation; anything beyond this is the hang. */
+    private static final long DEPLOY_TIMEOUT_SECONDS = 30;
+
+    private static final int ITERATIONS = 30;
+
+    @Test
+    public void deployMustNotHangWhenExecutionContextDisappearsDuringDeploy() throws Exception {
+        TaskExecutionService taskExecutionService = server.getTaskExecutionService();
+        ConcurrentMap<TaskGroupLocation, TaskGroupContext> executionContexts =
+                executionContextsOf(taskExecutionService);
+
+        ExecutorService deployer = Executors.newSingleThreadExecutor();
+        try {
+            for (int iteration = 0; iteration < ITERATIONS; iteration++) {
+                long jobId = System.nanoTime();
+                TaskGroupLocation location = new TaskGroupLocation(jobId, 1, 1);
+                AtomicBoolean stopTask = new AtomicBoolean(false);
+
+                // isThreadsShare() == false routes the task through submitBlockingTask(),
+                // which is the path that waits on startedLatch.
+                TestTask blockingTask = new TestTask(stopTask, 300, false);
+
+                TaskGroupImmutableInformation information =
+                        new TaskGroupImmutableInformation(
+                                jobId,
+                                1,
+                                TaskGroupType.INTERMEDIATE_BLOCKING_QUEUE,
+                                location,
+                                "staleContextRace",
+                                Collections.singletonList(
+                                        nodeEngine.getSerializationService().toData(blockingTask)),
+                                Collections.singletonList(emptySet()),
+                                Arrays.asList(emptySet()));
+                Data data = nodeEngine.getSerializationService().toData(information);
+
+                // Stand in for the previous generation's late taskDone(): clear the entry for
+                // this location while the deployment is in flight.
+                AtomicBoolean stopRemover = new AtomicBoolean(false);
+                Thread remover =
+                        new Thread(
+                                () -> {
+                                    while (!stopRemover.get()) {
+                                        executionContexts.remove(location);
+                                    }
+                                },
+                                "stale-taskDone-simulator");
+                remover.setDaemon(true);
+                remover.start();
+
+                Future<TaskDeployState> deployment =
+                        deployer.submit(() -> taskExecutionService.deployTask(data));
+                try {
+                    // Returning normally and throwing both satisfy the contract. Only a
+                    // deployment that never returns is the defect under test.
+                    deployment.get(DEPLOY_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+                } catch (ExecutionException expectedWhenContextIsGone) {
+                    // deployTask surfaced the missing context as an error - acceptable.
+                } catch (TimeoutException e) {
+                    deployment.cancel(true);
+                    Assertions.fail(
+                            "deployTask did not return within "
+                                    + DEPLOY_TIMEOUT_SECONDS
+                                    + "s on iteration "
+                                    + iteration
+                                    + ". A BlockingWorker that fails before"
+                                    + " startedLatch.countDown() leaves submitBlockingTask"
+                                    + " waiting forever while holding the SubPlan monitor."
+                                    + " See https://github.com/apache/seatunnel/issues/11679");
+                } finally {
+                    stopRemover.set(true);
+                    remover.join(TimeUnit.SECONDS.toMillis(5));
+                    stopTask.set(true);
+                    try {
+                        taskExecutionService.cancelTaskGroup(location);
+                    } catch (RuntimeException ignored) {
+                        // the group may already be gone; irrelevant to this assertion
+                    }
+                }
+            }
+        } finally {
+            deployer.shutdownNow();
+        }
+    }
+
+    @SuppressWarnings("unchecked")
+    private static ConcurrentMap<TaskGroupLocation, TaskGroupContext> executionContextsOf(
+            TaskExecutionService taskExecutionService) throws Exception {
+        Field field = TaskExecutionService.class.getDeclaredField("executionContexts");
+        field.setAccessible(true);
+        return (ConcurrentMap<TaskGroupLocation, TaskGroupContext>) field.get(taskExecutionService);
+    }
+}


(seatunnel) 01/07: [Fix][Zeta] Release the start latch when a blocking worker cannot start

Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit 4cbe31d1649d6f73a6634f08d244d7c3d4075663
Author: SAADAOUI ABDESSALEM <10...@users.noreply.github.com>
AuthorDate: Sun Aug 9 16:18:01 2026 +0200

    [Fix][Zeta] Release the start latch when a blocking worker cannot start
    
    BlockingWorker.run() resolved its class loader from executionContexts before its
    try block, while startedLatch.countDown() sat inside it. TaskGroupLocation is
    reused verbatim across pipeline restore generations, so a taskDone() from an
    earlier generation can remove the context the current deployLocalTask() has just
    installed. The worker then threw before signalling, the exception was swallowed
    by the submitting Future, and submitBlockingTask() waited on the latch forever
    while holding the SubPlan monitor - which prevented checkpoint error handling
    from moving the pipeline to a terminal state and left the seatunnel.sh client
    blocked indefinitely.
    
    Move the lookup inside the try, fail explicitly when the context is gone, and
    release the latch exactly once per worker on that path. The signal still happens
    before init(), so submitBlockingTask() still returns once workers have started
    rather than once they have finished.
    
    (cherry picked from commit b91e00582827218ca72013165f6d4c9a2800e0ab)
---
 .../engine/server/TaskExecutionService.java        | 37 ++++++++++++++++++----
 1 file changed, 31 insertions(+), 6 deletions(-)

diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
index eeef577ad2..c02d3fdea5 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
@@ -1067,17 +1067,36 @@ public class TaskExecutionService implements DynamicMetricsProvider {
         public void run() {
             TaskExecutionService.TaskGroupExecutionTracker taskGroupExecutionTracker =
                     tracker.taskGroupExecutionTracker;
-            ClassLoader classLoader =
-                    executionContexts
-                            .get(taskGroupExecutionTracker.taskGroup.getTaskGroupLocation())
-                            .getClassLoaders()
-                            .get(tracker.task.getTaskID());
+            TaskGroupLocation taskGroupLocation =
+                    taskGroupExecutionTracker.taskGroup.getTaskGroupLocation();
             ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
-            Thread.currentThread().setContextClassLoader(classLoader);
             final Task t = tracker.task;
             ProgressState result = null;
+            boolean startSignalled = false;
             try {
+                // Resolving the class loader has to happen inside this try. The execution
+                // context for this location can already have been removed by a taskDone() of
+                // an earlier restore generation - TaskGroupLocation is reused verbatim across
+                // restores - and dereferencing a missing context threw before startedLatch was
+                // ever counted down. submitBlockingTask() then waited on that latch forever
+                // while holding the SubPlan monitor, which stopped the pipeline from reaching a
+                // terminal state and left the submitting client blocked. See #11679.
+                TaskGroupContext taskGroupContext = executionContexts.get(taskGroupLocation);
+                if (taskGroupContext == null) {
+                    throw new IllegalStateException(
+                            String.format(
+                                    "Execution context for %s is no longer registered; the task"
+                                            + " group was cleaned up while it was being deployed",
+                                    taskGroupLocation));
+                }
+                Thread.currentThread()
+                        .setContextClassLoader(
+                                taskGroupContext.getClassLoaders().get(t.getTaskID()));
+
+                // Signalled before init() so that submitBlockingTask() returns once workers
+                // have started, not once they have finished.
                 startedLatch.countDown();
+                startSignalled = true;
                 t.init();
                 do {
                     result = t.call();
@@ -1098,6 +1117,12 @@ public class TaskExecutionService implements DynamicMetricsProvider {
                 }
                 taskGroupExecutionTracker.exception(e);
             } finally {
+                // A worker that failed before signalling must still release the deployer,
+                // otherwise submitBlockingTask() blocks forever. Guarded so each worker counts
+                // down exactly once.
+                if (!startSignalled) {
+                    startedLatch.countDown();
+                }
                 taskGroupExecutionTracker.taskDone(t);
                 if (result == null || !result.isDone()) {
                     try {


(seatunnel) 07/07: [Fix][Zeta] Rename startSignalled to startLatchReleased

Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit e6d5214c3cd24f2462e6ef5e088007781d243b75
Author: SAADAOUI ABDESSALEM <10...@users.noreply.github.com>
AuthorDate: Tue Aug 11 07:41:09 2026 +0200

    [Fix][Zeta] Rename startSignalled to startLatchReleased
    
    The old name read as 'has the task started', while it actually tracks whether
    this particular worker has already counted down its slot of the start latch.
    Rename only - no behaviour change.
    
    Suggested in review by @nzw921rx.
    
    (cherry picked from commit 79eebc5543408f8172945692b2a01acd24ab0546)
---
 .../org/apache/seatunnel/engine/server/TaskExecutionService.java    | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
index 0a278f21f3..a8c6b9893e 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
@@ -1074,7 +1074,7 @@ public class TaskExecutionService implements DynamicMetricsProvider {
             ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
             final Task t = tracker.task;
             ProgressState result = null;
-            boolean startSignalled = false;
+            boolean startLatchReleased = false;
             try {
                 // Resolving the class loader has to happen inside this try. The execution
                 // context for this location can already have been removed by a taskDone() of
@@ -1098,7 +1098,7 @@ public class TaskExecutionService implements DynamicMetricsProvider {
                 // Signalled before init() so that submitBlockingTask() returns once workers
                 // have started, not once they have finished.
                 startedLatch.countDown();
-                startSignalled = true;
+                startLatchReleased = true;
                 t.init();
                 do {
                     result = t.call();
@@ -1122,7 +1122,7 @@ public class TaskExecutionService implements DynamicMetricsProvider {
                 // A worker that failed before signalling must still release the deployer,
                 // otherwise submitBlockingTask() blocks forever. Guarded so each worker counts
                 // down exactly once.
-                if (!startSignalled) {
+                if (!startLatchReleased) {
                     startedLatch.countDown();
                 }
                 taskGroupExecutionTracker.taskDone(t);


(seatunnel) 06/07: [Test][Zeta] Assert the task group future settles on the stale-context path

Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit 85bdfc189f3041cc74e9c4a4be9272cb8d7c2a53
Author: SAADAOUI ABDESSALEM <10...@users.noreply.github.com>
AuthorDate: Mon Aug 10 14:58:57 2026 +0200

    [Test][Zeta] Assert the task group future settles on the stale-context path
    
    deployTask(Data) discards the future returned by deployLocalTask(), so the
    existing test can only observe that the deployment returned - which is already
    satisfied when startedLatch is released, before taskDone() runs. A failure
    inside taskDone() therefore leaves the task group's TaskExecutionState future
    pending forever while that test still passes.
    
    That future is what PhysicalVertex and SubPlan wait on to drive pipeline state,
    so never completing it reproduces a hang of the same class as #11679 one level
    up. The new test calls deployLocalTask() directly to hold the future and keeps
    the remover racing until after it has been observed. Completing exceptionally
    satisfies the contract; only never completing is the defect.
    
    Verified on the JDK 8 leg against the same commit, test unchanged in both:
    with the taskDone guard reverted the new method fails on iteration 0 while the
    existing method still passes; with the guard restored both pass in 4.5s.
    
    Raised as Issue 3 in review.
    
    (cherry picked from commit e9b25d84d55f6b0afcf27444c12858ef3a93cdaa)
---
 .../server/TaskDeployStaleContextRaceTest.java     | 111 ++++++++++++++++++++-
 1 file changed, 109 insertions(+), 2 deletions(-)

diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
index 22b808dee6..4289aed6c6 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
@@ -17,8 +17,12 @@
 
 package org.apache.seatunnel.engine.server;
 
+import org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture;
+import org.apache.seatunnel.engine.server.execution.Task;
 import org.apache.seatunnel.engine.server.execution.TaskDeployState;
+import org.apache.seatunnel.engine.server.execution.TaskExecutionState;
 import org.apache.seatunnel.engine.server.execution.TaskGroupContext;
+import org.apache.seatunnel.engine.server.execution.TaskGroupDefaultImpl;
 import org.apache.seatunnel.engine.server.execution.TaskGroupLocation;
 import org.apache.seatunnel.engine.server.execution.TaskGroupType;
 import org.apache.seatunnel.engine.server.execution.TestTask;
@@ -30,8 +34,11 @@ import org.junit.jupiter.api.Test;
 import com.hazelcast.internal.serialization.Data;
 
 import java.lang.reflect.Field;
+import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentMap;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
@@ -56,13 +63,13 @@ import static java.util.Collections.emptySet;
  * executionContexts.get(location)} <em>before</em> its {@code try} block, and {@code
  * startedLatch.countDown()} sits <em>inside</em> it. A missing context therefore throws before the
  * latch is ever counted down, the exception is swallowed by the submitting {@code Future}, and
- * {@code submitBlockingTask()} waits on {@code startedLatch.await()} forever — while holding the
+ * {@code submitBlockingTask()} waits on {@code startedLatch.await()} forever - while holding the
  * {@code SubPlan} monitor, which in turn blocks checkpoint-error handling from ever moving the
  * pipeline to a terminal state.
  *
  * <p>This test asserts the narrow contract that prevents the hang: <b>deploying a task group must
  * return, even if the execution context for that location disappears while the deployment is in
- * flight.</b> Whether it returns successfully or throws is not asserted — only that it does not
+ * flight.</b> Whether it returns successfully or throws is not asserted - only that it does not
  * block indefinitely.
  *
  * <p>The race window is small, so the test repeats the deployment and races a remover thread
@@ -160,6 +167,106 @@ public class TaskDeployStaleContextRaceTest extends AbstractSeaTunnelServerTest
         }
     }
 
+    /**
+     * The sibling contract to the test above, on the other side of the same race.
+     *
+     * <p>{@code deployTask(Data)} discards the future returned by {@code deployLocalTask()}, so a
+     * test driven through it can only observe that the deployment returned - which is satisfied the
+     * moment {@code startedLatch} is released, before {@code taskDone()} runs. A failure inside
+     * {@code taskDone()} therefore leaves the task group's completion future pending forever while
+     * that test still passes.
+     *
+     * <p>That future is the one {@code PhysicalVertex} and {@code SubPlan} wait on to drive
+     * pipeline state, so never completing it reproduces a hang of the same class as the one this
+     * class is named for, one level up. This test calls {@code deployLocalTask()} directly to hold
+     * the future, and keeps the remover racing until after it has been observed.
+     *
+     * <p>Completing exceptionally satisfies the contract; only never completing is the defect.
+     */
+    @Test
+    public void taskGroupFutureMustCompleteWhenExecutionContextDisappearsDuringDeploy()
+            throws Exception {
+        TaskExecutionService taskExecutionService = server.getTaskExecutionService();
+        ConcurrentMap<TaskGroupLocation, TaskGroupContext> executionContexts =
+                executionContextsOf(taskExecutionService);
+
+        ExecutorService deployer = Executors.newSingleThreadExecutor();
+        try {
+            for (int iteration = 0; iteration < ITERATIONS; iteration++) {
+                long jobId = System.nanoTime();
+                TaskGroupLocation location = new TaskGroupLocation(jobId, 1, 1);
+                AtomicBoolean stopTask = new AtomicBoolean(false);
+
+                TestTask blockingTask = new TestTask(stopTask, 300, false);
+                List<Task> tasks = new ArrayList<>();
+                tasks.add(blockingTask);
+                TaskGroupDefaultImpl taskGroup =
+                        new TaskGroupDefaultImpl(location, "staleContextRaceFuture", tasks);
+
+                ConcurrentHashMap<Long, ClassLoader> classLoaders = new ConcurrentHashMap<>();
+                classLoaders.put(
+                        blockingTask.getTaskID(), Thread.currentThread().getContextClassLoader());
+
+                AtomicBoolean stopRemover = new AtomicBoolean(false);
+                Thread remover =
+                        new Thread(
+                                () -> {
+                                    while (!stopRemover.get()) {
+                                        executionContexts.remove(location);
+                                    }
+                                },
+                                "stale-taskDone-simulator");
+                remover.setDaemon(true);
+                remover.start();
+
+                try {
+                    // Deployment itself is covered by the test above; it runs on a separate
+                    // thread here only so a regression there cannot hang this one too.
+                    Future<PassiveCompletableFuture<TaskExecutionState>> deployment =
+                            deployer.submit(
+                                    () ->
+                                            taskExecutionService.deployLocalTask(
+                                                    taskGroup,
+                                                    classLoaders,
+                                                    new ConcurrentHashMap<>()));
+                    PassiveCompletableFuture<TaskExecutionState> completion =
+                            deployment.get(DEPLOY_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+
+                    // Let the task finish on the iterations where the context survived, so
+                    // the group completes for the same reason in both branches.
+                    stopTask.set(true);
+
+                    try {
+                        completion.get(DEPLOY_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+                    } catch (ExecutionException completedExceptionally) {
+                        // Still completed. Only a future that never settles is the defect.
+                    } catch (TimeoutException e) {
+                        Assertions.fail(
+                                "The task group future did not complete within "
+                                        + DEPLOY_TIMEOUT_SECONDS
+                                        + "s on iteration "
+                                        + iteration
+                                        + ". A failure inside taskDone() before"
+                                        + " future.complete() leaves PhysicalVertex and SubPlan"
+                                        + " waiting on a future that never settles."
+                                        + " See https://github.com/apache/seatunnel/issues/11679");
+                    }
+                } finally {
+                    stopRemover.set(true);
+                    remover.join(TimeUnit.SECONDS.toMillis(5));
+                    stopTask.set(true);
+                    try {
+                        taskExecutionService.cancelTaskGroup(location);
+                    } catch (RuntimeException ignored) {
+                        // the group may already be gone; irrelevant to this assertion
+                    }
+                }
+            }
+        } finally {
+            deployer.shutdownNow();
+        }
+    }
+
     @SuppressWarnings("unchecked")
     private static ConcurrentMap<TaskGroupLocation, TaskGroupContext> executionContextsOf(
             TaskExecutionService taskExecutionService) throws Exception {


(seatunnel) 03/07: [Fix][Zeta] Guard recycleClassLoader against a missing execution context

Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit e8e1eb7a725b957b70d988f08920007352fe95e5
Author: SAADAOUI ABDESSALEM <10...@users.noreply.github.com>
AuthorDate: Mon Aug 10 10:30:37 2026 +0200

    [Fix][Zeta] Guard recycleClassLoader against a missing execution context
    
    taskDone() is now reached on every exit from BlockingWorker.run(), so the
    cross-generation removal race that motivates this PR is more likely than before
    to reach recycleClassLoader() after the context has already been removed. The
    unguarded lookup there would throw from a finally block and skip the caller's
    context class loader restore, leaving a stale loader on a pooled thread.
    
    Skip the release with a warning when the context is gone, and drop the
    redundant second map lookup while touching it.
    
    Raised as Issue 1 in review.
    
    (cherry picked from commit 2e9d605706678194a4b7ee9be1d1406f79405b91)
---
 .../seatunnel/engine/server/TaskExecutionService.java | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
index c02d3fdea5..f89f18c1ba 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/TaskExecutionService.java
@@ -1055,11 +1055,13 @@ public class TaskExecutionService implements DynamicMetricsProvider {
          * <p>Execution flow:
          *
          * <ol>
-         *   <li>Set up the class loader for the task
+         *   <li>Resolve the task group's execution context and set up the class loader
          *   <li>Signal that the worker has started via CountDownLatch
          *   <li>Initialize the task via {@link Task#init()}
          *   <li>Execute the task repeatedly via {@link Task#call()} until done
          *   <li>Handle interrupts and exceptions, notifying the execution tracker
+         *   <li>Release the start latch anyway if the worker failed before signalling, so the
+         *       deployer waiting in {@code submitBlockingTask()} is never left blocked
          *   <li>Clean up by calling {@link Task#close()} if not completed
          * </ol>
          */
@@ -1469,7 +1471,20 @@ public class TaskExecutionService implements DynamicMetricsProvider {
 
         private void recycleClassLoader(TaskGroupLocation taskGroupLocation) {
             TaskGroupContext context = executionContexts.get(taskGroupLocation);
-            executionContexts.get(taskGroupLocation).setClassLoaders(null);
+            if (context == null) {
+                // The same cross-generation race that motivates this fix can remove the
+                // context before the last task of the group finishes. taskDone() is now
+                // reached on every exit from BlockingWorker.run(), so this path is more
+                // likely than it was; without the guard it would throw from a finally
+                // block and skip the caller's context class loader restore.
+                logger.warning(
+                        String.format(
+                                "Execution context for %s is no longer registered; skipping class"
+                                        + " loader release",
+                                taskGroupLocation));
+                return;
+            }
+            context.setClassLoaders(null);
             for (Collection<URL> jars : context.getJars().values()) {
                 classLoaderService.releaseClassLoader(taskGroupLocation.getJobId(), jars);
             }


(seatunnel) 04/07: [Test][Zeta] Clarify the review points on the regression test and Javadoc

Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

davidzollo pushed a commit to branch davidzollo_republish_11727_start_latch_release
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit fc758e305480b95afc95f28d93e5a9b52a35fec0
Author: SAADAOUI ABDESSALEM <10...@users.noreply.github.com>
AuthorDate: Mon Aug 10 10:30:38 2026 +0200

    [Test][Zeta] Clarify the review points on the regression test and Javadoc
    
    - BlockingWorker.run() javadoc now lists the failure-path latch release.
    - The test's ExecutionException branch is documented as defensive only:
      deployTask() wraps its body in a catch-all and returns TaskDeployState.failed,
      and a worker's failure is recorded on the tracker from another thread, so that
      branch is not an expected path.
    
    Raised as Issues 2 and 3 in review.
    
    (cherry picked from commit 8cbe828b22181642f13205be2e51b0f7b4235187)
---
 .../engine/server/TaskDeployStaleContextRaceTest.java        | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
index b3b7357fc4..22b808dee6 100644
--- a/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
+++ b/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/TaskDeployStaleContextRaceTest.java
@@ -124,11 +124,15 @@ public class TaskDeployStaleContextRaceTest extends AbstractSeaTunnelServerTest
                 Future<TaskDeployState> deployment =
                         deployer.submit(() -> taskExecutionService.deployTask(data));
                 try {
-                    // Returning normally and throwing both satisfy the contract. Only a
-                    // deployment that never returns is the defect under test.
+                    // Returning at all is what matters. Only a deployment that never
+                    // returns is the defect under test.
                     deployment.get(DEPLOY_TIMEOUT_SECONDS, TimeUnit.SECONDS);
-                } catch (ExecutionException expectedWhenContextIsGone) {
-                    // deployTask surfaced the missing context as an error - acceptable.
+                } catch (ExecutionException defensiveOnly) {
+                    // Not an expected path: deployTask() wraps its whole body in a
+                    // catch-all and returns TaskDeployState.failed(t) rather than
+                    // throwing, and a BlockingWorker's own failure is recorded on the
+                    // execution tracker from another thread. Caught only so an
+                    // unforeseen wrapper exception cannot be mistaken for a hang.
                 } catch (TimeoutException e) {
                     deployment.cancel(true);
                     Assertions.fail(