[I] Add end-to-end integration test for multi-region workflow execution [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

aglinxinyuan opened a new issue, #6133:
URL: https://github.com/apache/texera/issues/6133

   ### Task Summary
   
   Cross-region data delivery goes through **input-port materialization reader threads** (which replaced the cache-source operators in #3425). This path has **no end-to-end coverage** — it is exercised only by scheduling unit tests that count regions on a physical plan, so a regression in the materialization read/write path could pass CI unnoticed.
   
   This matters right now because the now-dead cache-source plumbing is about to be removed (#6125 / #6126). The multi-region path — exactly what the removed cache sources used to serve — should get end-to-end coverage **before** that refactor.
   
   Add an integration test that drives a big "X"-shaped workflow across multiple regions:
   
   ```
   csvLeft  ─┬─▶ join.build (port 0) ─┐
             │                         join ─▶ pythonSort ─▶ (terminal)
   csvRight ─┼─▶ join.probe (port 1) ─┘
             │
   csvLeft  ─┼─▶ union ───────────────────────────────────▶ (terminal)
   csvRight ─┘   (two links fan into union's single port)
   ```
   
   The hash join's probe-depends-on-build ordering forces the build input to be materialized, cutting the plan into ≥2 regions whose boundary is crossed by the reader-thread path. A Python sort UDF makes it a real end-to-end run with worker subprocesses.
   
   Assertions:
   
   | Aspect | Expectation |
   | --- | --- |
   | Region count | plan spans **≥2 regions** (via the same `CostBasedScheduleGenerator` the controller uses at runtime) |
   | Join branch | self-join on the unique `Order ID` key → **100 rows**, returned **sorted** by the Python UDF |
   | Union branch | concatenation of both sources → **200 rows** |
   
   Notes:
   - Python-tagged (`@IntegrationTest`) so it runs in the `amber-integration` CI job (which provisions Python deps); the lighter `amber` job excludes it.
   - Should land **before** #6126, which is then rebased on top so its CI re-runs this test against the removal.
   
   ### Task Type
   
   - [x] Test / Coverage
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add end-to-end integration test for multi-region workflow execution [texera]

["aglinxinyuan (via GitHub)" <gi...@apache.org>]

aglinxinyuan closed issue #6133: Add end-to-end integration test for multi-region workflow execution
URL: https://github.com/apache/texera/issues/6133


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add end-to-end integration test for multi-region workflow execution [texera]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #6133:
URL: https://github.com/apache/texera/issues/6133#issuecomment-4884677184

   <!-- texera:template-compliance -->
   👋 Thanks for opening this issue, @aglinxinyuan!
   
   It looks like the issue description doesn't quite follow our template yet:
   
   - This issue doesn't appear to use one of our templates (Bug, Feature, or Task). Please open it using a template so the required details are captured.
   
   Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed.
   
   You can find the template prompts by editing the description, or see [CONTRIBUTING.md](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) for the full contribution flow.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org