You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "CRZbulabula (via GitHub)" <gi...@apache.org> on 2026/07/15 04:50:27 UTC

[PR] Fix procedures stuck after submission-time deserialization failure [iotdb]

CRZbulabula opened a new pull request, #18212:
URL: https://github.com/apache/iotdb/pull/18212

   ## Description
   
   ### Problem
   
   Persisting a newly submitted procedure can serialize and deserialize it through the consensus layer. If deserialization marks the procedure as `FAILED` before it is added to the executor, the root rollback stack is empty. The worker then skips execution because the procedure is not `RUNNABLE`, but the procedure remains in the active map. `waitingProcedureFinished()` eventually reports `OVERLAP_WITH_EXISTING_TASK`, causing the DataNode to retry indefinitely and leaving the CLI request blocked.
   
   ### Fix
   
   Initialize the root rollback stack with a newly submitted procedure when it is already failed. This lets the executor follow its normal rollback and cleanup path, transition the procedure to `ROLLEDBACK`, remove it from the active map, and expose the original failure to the caller.
   
   ### Tests
   
   Added a unit test that simulates the first procedure-store update marking the procedure as failed and verifies that:
   
   - `execute()` is not invoked.
   - `rollback()` is invoked exactly once.
   - the procedure reaches `ROLLEDBACK`.
   - the executor removes the procedure from its active map.
   
   Validation:
   
   - `mvn spotless:apply -pl iotdb-core/confignode`
   - `mvn test -pl iotdb-core/confignode -Dtest=TestProcedureExecutor -DfailIfNoTests=false`
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the why and intent of the code.
   - [x] added or updated unit tests to cover the new code path.
   
   <hr>
   
   ##### Key changed/added classes
   
   - `ProcedureExecutor`
   - `TestProcedureExecutor`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix procedures stuck after submission-time deserialization failure [iotdb]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18212:
URL: https://github.com/apache/iotdb/pull/18212#discussion_r3584800127


##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java:
##########
@@ -810,6 +810,13 @@ private long pushProcedure(Procedure<Env> procedure) {
     // Update metrics on start of a procedure
     procedure.updateMetricsOnSubmit(getEnvironment());
     RootProcedureStack<Env> stack = new RootProcedureStack<>();
+    // Persisting a newly submitted procedure may serialize and deserialize it through the
+    // consensus layer. If that process marks the procedure as failed before it is scheduled, the
+    // rollback stack still needs an entry so the executor can finish the failed procedure instead
+    // of leaving it in the active procedure map forever.
+    if (procedure.isFailed()) {
+      stack.addRollbackStep(procedure);
+    }

Review Comment:
   `pushProcedure()` adds a rollback step when the procedure is already FAILED, but that also mutates the procedure’s `stackIndexes` (via `addRollbackStep`). Since this happens after the initial `store.update()` in `submitProcedure()`, the rollback step isn’t persisted. If the ConfigNode leader crashes/restarts before the rollback completes, recovery will load a FAILED root procedure with no stack and `executeRootStackRollback()` will hit an NPE when it assumes `RootProcedureStack.getSubproceduresStack()` is non-null.
   
   Consider persisting the updated failed procedure (with the initialized rollback stack entry) immediately after adding the rollback step so recovery can safely roll it back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix procedures stuck after submission-time deserialization failure [iotdb]

Posted by "CRZbulabula (via GitHub)" <gi...@apache.org>.
CRZbulabula commented on code in PR #18212:
URL: https://github.com/apache/iotdb/pull/18212#discussion_r3585329186


##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/ProcedureExecutor.java:
##########
@@ -810,6 +810,13 @@ private long pushProcedure(Procedure<Env> procedure) {
     // Update metrics on start of a procedure
     procedure.updateMetricsOnSubmit(getEnvironment());
     RootProcedureStack<Env> stack = new RootProcedureStack<>();
+    // Persisting a newly submitted procedure may serialize and deserialize it through the
+    // consensus layer. If that process marks the procedure as failed before it is scheduled, the
+    // rollback stack still needs an entry so the executor can finish the failed procedure instead
+    // of leaving it in the active procedure map forever.
+    if (procedure.isFailed()) {
+      stack.addRollbackStep(procedure);
+    }

Review Comment:
   Addressed in 8b9589b5f0. The submission path now persists the failed procedure again after initializing its rollback entry. Recovery also reconstructs a missing rollback entry for a FAILED procedure with no stack indexes, covering the crash window before that second persistence completes and any previously stored affected procedure. Added regression coverage for both persistence-before-scheduling and restart recovery.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix procedures stuck after submission-time deserialization failure [iotdb]

Posted by "CRZbulabula (via GitHub)" <gi...@apache.org>.
CRZbulabula merged PR #18212:
URL: https://github.com/apache/iotdb/pull/18212


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org