[I] A worker reading a materialized input can miss its port completion and hang the region [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

aglinxinyuan opened a new issue, #8937:
URL: https://github.com/apache/texera/issues/8937

   ### What happened?
   
   Two `LoopIntegrationSpec` cases added in #8639 time out intermittently on `main` and on unrelated PRs (macOS and Ubuntu `amber-integration` jobs):
   
   - "bind the inner loop's variable where it shadows the outer loop's of the same name"
   - "bind a $i reference after two branches of the loop body meet again"
   
   Each failure is `com.twitter.util.TimeoutException: 1.minutes+30.seconds` at about 180 s: the 90 s deadline, once per `withRetry` attempt. Codecov test analytics lists about 10 such failures on `main` between 2026-10-05 and 2026-10-07. The older loop cases have failed the same way now and then, for example "run a nested loop whose inner body chains multiple JVM operators" on `main` on 2026-08-25.
   
   These runs are not just slow. Locally the two cases take 55 s and 31 s, and a timed-out run is idle: the JVM's CPU is flat and every dispatcher is waiting. One worker of the 2-worker Filter (usually the one that got no tuples) handles the materialized input's END_CHANNEL and reports its input port completed. After that it never finalizes its output and never reports that it finished, so the region never completes. Nothing is logged.
   
   The cause is a race between `InputPortMaterializationReaderThread` and the DP thread:
   
   - The reader thread enqueues END_CHANNEL, and only after that sets its `finished` flag.
   - `EndChannelHandler` finalizes the worker's output only when `InputManager.isPortCompleted` holds for every input port. For a port read from materialization, that method returns the reader thread's `finished` flag.
   - A DP thread that is waiting on the queue can take END and run `EndChannelHandler` before the reader thread sets the flag. It sees the port as not completed and skips `finalizeOutput`, and nothing calls it again.
   
   ```
   Before:  reader enqueues END -> DP processes END, flag not set yet -> output never finalized -> region hangs
   After:   reader enqueues END -> DP processes END, port completed   -> output finalized       -> region completes
   ```
   
   The same flag also goes wrong in the other direction. It can already be set while END is still queued, so a worker with a second materialized input port could finalize before that port's END has been processed.
   
   The race dates from #3460 (June 2025), which added the flag and this check. #8639's cases hit it most often because they re-run a 2-worker JVM operator fed from a materialized input on every loop iteration. Expected behavior: a materialized input port counts as completed when its END has been processed, the same as a port fed by a link.
   
   ### How to reproduce?
   
   On `main`, pin the sbt JVM to 3 CPUs (its Python workers inherit the affinity), then run the branching case a few times:
   
   ```
   sbt "WorkflowExecutionService/testOnly *LoopIntegrationSpec -- -z \"after two branches of the loop body meet again\""
   ```
   
   With `UDF_PYTHON_PATH`, `STORAGE_ICEBERG_CATALOG_TYPE=postgres` and `TEXERA_HOME` set, and `AMBER_TEST_FILTER` unset so that `-z` selects the case. On a 32-core Windows machine, 3 of 5 pinned runs hung on the first attempt. One of them hung on the retry too and failed exactly as on CI. A healthy pinned run takes 40 s. Without pinning, all 8 cases pass, in 26–55 s each.
   
   For a deterministic repro, call `EndChannelHandler.endChannel` on a worker whose only input port is read from materialization, before the reader thread sets its flag. The worker emits only `FinalizePort(input)`, with no output `FinalizePort` and no `FinalizeExecutor`.
   
   ### Version/Branch
   
   1.4.0-incubating-SNAPSHOT (main)
   
   ### Commit Hash (Optional)
   
   be65bf8564
   
   ### What browsers are you seeing the problem on?
   
   _No response_
   
   ### Relevant log output
   
   ```shell
   com.twitter.util.TimeoutException: 1.minutes+30.seconds
   	at com.twitter.util.Promise.ready(Promise.scala:680)
   	at com.twitter.util.Await$.result(Awaitable.scala:155)
   	at org.apache.texera.amber.engine.e2e.TestUtils$.runWorkflowAndReadResults(TestUtils.scala:187)
   	at org.apache.texera.amber.engine.e2e.LoopIntegrationSpec.runAndGetMaterializedRowCounts(LoopIntegrationSpec.scala:163)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] A worker reading a materialized input can miss its port completion and hang the region [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

aglinxinyuan closed issue #8937: A worker reading a materialized input can miss its port completion and hang the region
URL: https://github.com/apache/texera/issues/8937


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org