You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org> on 2026/04/29 17:28:25 UTC

[I] Make region termination synchronous before scheduling the next region [texera]

Xiao-zhen-Liu opened a new issue, #4556:
URL: https://github.com/apache/texera/issues/4556

   ### What happened?
   
   Region execution coordination can schedule the next region before the previous region's workers are fully terminated.
   
   This can lead to unsafe interleavings between region termination, next-region startup, and workflow completion. In particular, when `endWorker` is sent while a worker still has queued messages, the worker rejects termination with errors like:
   
   `Received EndHandler before all messages are processed`
   
   In the synchronous kill path, workflow completion can also be emitted during the gap where the previous initialized region is completed but the next pending region has not been initialized yet.
   
   The coordinator should not start the next region until the previous region's workers have acknowledged `endWorker` and have been gracefully stopped.
   
   If `endWorker` fails because the worker still has queued messages, termination should retry until the worker queue is drained and `endWorker` succeeds. Workflow completion should only be emitted after all scheduled regions have finished.
   
   
   ### How to reproduce?
   
   Run a workflow with multiple scheduled regions where the first region completes while workers may still have queued control messages, such as statistics query replies or other control messages.
   
   Steps:
   1. Start a workflow that is split into at least two regions.
   2. Let the first region complete.
   3. Observe that the coordinator can attempt to start the second region before the first region's worker termination is fully complete.
   4. In some runs, observe worker termination failures like:
      `Received EndHandler before all messages are processed`
   5. In some runs, observe premature workflow completion before the next region has fully launched or finished.
   
   
   ### Version
   
   1.1.0-incubating (Pre-release/Master)
   
   ### Commit Hash (Optional)
   
   _No response_
   
   ### What browsers are you seeing the problem on?
   
   _No response_
   
   ### Relevant log output
   
   ```shell
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Make region termination synchronous before scheduling the next region [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #4556:
URL: https://github.com/apache/texera/issues/4556#issuecomment-4346811738

   hit: @Xiao-zhen-Liu can you change the title and description as a bug? what doesn't work and what affects users? the current title "Make region termination synchronous before scheduling the next region" is not describing a bug, but more like a solution or a task. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Make region termination synchronous before scheduling the next region [texera]

Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on issue #4556:
URL: https://github.com/apache/texera/issues/4556#issuecomment-4346358969

   In general, we need to manage the lifecycle of each region and its workers carefully.  We should let the current region finish its execution completely before starting the next region.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Workers in a previous region are not fully terminated before the next region is started [texera]

Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan closed issue #4556: Workers in a previous region are not fully terminated before the next region is started
URL: https://github.com/apache/texera/issues/4556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Workers in a previous region are not fully terminated before the next region is started [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #4556:
URL: https://github.com/apache/texera/issues/4556#issuecomment-4347217965

   Thanks. As mentioned in the PR #4557. The current assessment of this bug is low priority as it does not affect user and it only blocks control block work. Feel free to re-evaluate it. 
   
   In general I suggest we fix it those low priority ones after the release. because every "fix" could also introduce new bugs, which needs time to discover and fix, which may delay release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org