[D] Loop Start requires exactly one materialized reader on its input port, is lifting this planned? [texera]

["carloea2 (via GitHub)" <gi...@apache.org>]

GitHub user carloea2 created a discussion: Loop Start requires exactly one materialized reader on its input port, is lifting this planned?

## Context

I build Texera `WorkflowContent` programmatically and run it on a current dev build. Some of the
plans I generate have a Loop Start whose single input port is fed by **two upstream operators**
(two different producers each contribute values the loop body reads). Those plans are rejected at
`StartWorkflow`:

```
java.lang.IllegalArgumentException: requirement failed:
Loop Start input port GlobalPortIdentity(
    PhysicalOpIdentity(OperatorIdentity(LoopStart-operator-...),main),
    PortIdentity(0,false), true)
  expected exactly one reader URI, got 2
    at WorkflowExecutionManager.$anonfun$loopStartStateUris$3(WorkflowExecutionManager.scala:86)
    at WorkflowExecutionManager.loopStartStateUris(WorkflowExecutionManager.scala:69)
    at WorkflowExecutionManager.advanceRegionExecutions(WorkflowExecutionManager.scala:131)
    at StartWorkflowHandler.startWorkflow(StartWorkflowHandler.scala:45)
```

## Restriction

The check is in `loopStartStateUris`
(`amber/.../scheduling/WorkflowExecutionManager.scala`):

```scala
require(
  cfg.storagePairs.size == 1,
  s"Loop Start input port $gpid expected exactly one reader URI, got ${cfg.storagePairs.size}"
)
op.id.logicalOpId.id -> VFSURIFactory.stateURI(cfg.storagePairs.head._1).toString
```
## Questions

1. **Is the single-reader restriction intended to be permanent**, or a placeholder until loop input
   fan-in is supported? 
2. **Is per-port channel accounting on the roadmap** completing a port only when all of its
   channels have ended, and firing `produce_state_on_finish` / `on_finish` / `PortCompletedRequest`
   once at that point? That looks like the change that would make loop fan-in fall out for free, but
   it also moves when regions advance, so we assume it is deliberate that it works the way it does.

GitHub link: https://github.com/apache/texera/discussions/6966

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org

Re: [D] Loop Start requires exactly one materialized reader on its input port, is lifting this planned? [texera]

["carloea2 (via GitHub)" <gi...@apache.org>]

GitHub user carloea2 added a comment to the discussion: Loop Start requires exactly one materialized reader on its input port, is lifting this planned?

Why is this the design?
Thanks

GitHub link: https://github.com/apache/texera/discussions/6966#discussioncomment-17808265

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org

Re: [D] Loop Start requires exactly one materialized reader on its input port, is lifting this planned? [texera]

["chenlica (via GitHub)" <gi...@apache.org>]

GitHub user chenlica added a comment to the discussion: Loop Start requires exactly one materialized reader on its input port, is lifting this planned?

To chime in, there are two possible ways to "combine" the tables of different iterations in a "ForLoop":

1) Produce a union of these tables;
2) Produce a table of these tables.

Currently our design is 1).  We can consider 2) in the future if there are use cases.

GitHub link: https://github.com/apache/texera/discussions/6966#discussioncomment-17873056

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org

Re: [D] Loop Start requires exactly one materialized reader on its input port, is lifting this planned? [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

GitHub user aglinxinyuan added a comment to the discussion: Loop Start requires exactly one materialized reader on its input port, is lifting this planned?

For each iteration, the LoopStart Operator reads the input data table and emits it downstream. Because the LoopStart Operator is restarted for every iteration, it must reload the input data table from the upstream materialization. To keep the implementation simple, we currently support only a single upstream materialization URI. In the future, this can be optimized by keeping the LoopStart Operator alive across iterations instead of restarting it each time.


GitHub link: https://github.com/apache/texera/discussions/6966#discussioncomment-17808498

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org

Re: [D] Loop Start requires exactly one materialized reader on its input port, is lifting this planned? [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

GitHub user aglinxinyuan added a comment to the discussion: Loop Start requires exactly one materialized reader on its input port, is lifting this planned?

It's by design. You need a union to join them.

GitHub link: https://github.com/apache/texera/discussions/6966#discussioncomment-17808236

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org

Re: [D] Loop Start requires exactly one materialized reader on its input port, is lifting this planned? [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

GitHub user aglinxinyuan added a comment to the discussion: Loop Start requires exactly one materialized reader on its input port, is lifting this planned?

I'll look into removing this limitation soon. For now, you can use Union as a workaround.

GitHub link: https://github.com/apache/texera/discussions/6966#discussioncomment-17808631

----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org