[I] Loop cannot run when a Java/Scala operator is inside the loop body [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

aglinxinyuan opened a new issue, #6660:
URL: https://github.com/apache/texera/issues/6660

   ### What happened?
   
   A loop cannot run when a Java/Scala built-in operator (e.g. `Limit`) is inside the loop body. The Loop End worker crashes on the first iteration:
   
   ```
   RuntimeError: no loop-back state URI configured for LoopStart ''
     (have: ['LoopStart-operator-4d423fa9-...'])
   ```
   
   The worker enters an error pause and the loop never iterates. A Python-only loop body works fine.
   
   ### How to reproduce?
   
   Run any loop with a Scala operator between Loop Start and Loop End, e.g.:
   
   ```
   TextInput ──▶ LoopStart ──▶ Limit ──▶ LoopEnd
                     ▲          (Scala)     │
                     └──── back-edge ───────┘
   ```
   
   ### Root cause
   
   Per-iteration loop state rides a `StateFrame` envelope — `loop_counter` (nesting depth countdown) and `loop_start_id` (the back-jump target) — materialized as their own columns next to `content` (`State` schema, #5900). The **Python** worker carries the envelope on every hop. The **Scala** side strips it at every seam: `StateFrame` has no envelope fields, so a JVM hop rebuilds states with the defaults `(0, "")`.
   
   | Seam (Scala) | What it does today |
   |---|---|
   | `StateFrame` (`DataPayload.scala`) | no `loopCounter` / `loopStartId` fields |
   | `InputPortMaterializationReaderThread` | rebuilds via `State.fromTuple(row)` — columns dropped |
   | `DataProcessor.processInputState` | forwards `processState` output without the incoming envelope |
   | `OutputManager.saveStateToStorageIfNeeded` | writes `state.toTuple()` — defaults `(0, "")` |
   | `NetworkOutputBuffer.sendState` | builds `StateFrame(state)` — defaults |
   | `PythonProxyClient` / `PythonProxyServer` | Arrow flight bridge drops the columns in both directions |
   
   So `LoopStart → Limit → LoopEnd`: LoopStart stamps `(0, LoopStart-id)` → the `Limit` hop re-emits `(0, "")` → LoopEnd captures `loop_start_id = ""` → the back-jump write-address lookup (`InitializeExecutorRequest.loopStartStateUris`) fails with the error above.
   
   In a **nested** loop the zeroed `loop_counter` is just as harmful: an enclosing loop's state crossing a Scala hop inside the inner body loses its depth, so the inner Loop End mis-consumes it instead of passing it through.
   
   ### Expected behavior
   
   | | before | after |
   |---|---|---|
   | `LoopStart → Limit → LoopEnd` (3 iters) | worker error, 0 iterations | 3 iterations, LoopEnd materializes 3 rows |
   | nested 3×3 with `Limit` in inner body | inner LoopEnd mis-consumes outer state | 9 outer rows / 3 inner rows |
   
   The envelope columns already exist in the materialized State schema and on the Arrow flight wire — a JVM operator only ever needs to **carry them through unchanged** (loop operators are Python-only; the +1/−1 bookkeeping lives in the Python runtime).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Loop cannot run when a Java/Scala operator is inside the loop body [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

aglinxinyuan closed issue #6660: Loop cannot run when a Java/Scala operator is inside the loop body
URL: https://github.com/apache/texera/issues/6660


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Loop cannot run when a Java/Scala operator is inside the loop body [texera]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #6660:
URL: https://github.com/apache/texera/issues/6660#issuecomment-5029046388

   <!-- texera:template-compliance -->
   👋 Thanks for opening this issue, @aglinxinyuan!
   
   It looks like the issue description doesn't quite follow our template yet:
   
   - This issue doesn't appear to use one of our templates (Bug, Feature, or Task). Please open it using a template so the required details are captured.
   
   Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed.
   
   You can find the template prompts by editing the description, or see [CONTRIBUTING.md](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) for the full contribution flow.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org