You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org> on 2026/09/28 19:21:54 UTC
[PR] feat(amber): schedule saved-result reads for cache reuse [texera]
Xiao-zhen-Liu opened a new pull request, #8732:
URL: https://github.com/apache/texera/pull/8732
### What changes were proposed in this PR?
This PR is part of cache reuse (#5881), where a run reads the saved results of an earlier run instead of computing them again. It lets the caller of `CostBasedScheduleGenerator` say, before scheduling, that some input ports read saved results, and where those results are stored. When part of a run is skipped, the operators that still run read the skipped operators' outputs this way. With no such input, which is every caller on main, the schedule is exactly as before.
- `CacheReadInputs` maps input ports to the storage locations they read (`readerUris`). It is a new last parameter of `CostBasedScheduleGenerator` and `ScheduleGenerator`, empty by default.
- In its second pass, the generator adds these locations to the ones an input port gets from materialized links. A port can get both: for example a Union with one input from a saved result and one from an operator in an earlier region. When the other input is a pipelined link, its rows arrive over the link, and only the saved result is read from storage.
- `DefaultResourceAllocator` is told which operators read saved results. In the plan it is given, such an operator can have no upstream link, so it would count as a source and use its own input requirement (say, a hash partition) as its output partitioning; the next link would then get a shuffle it never gets in the whole plan. It is partitioned like an operator behind a materialized link instead.
- The plan given to the generator for a partly skipped run leaves out the links that read saved results, but operators keep their own link lists, which still name those links. So `Region.getSourceOperators` and the reconfiguration scope in `FriesReconfigurationAlgorithm` now use only the region's own links. On main every link between two operators of a region is one of its links, so their answers do not change.
### Any related issues, documentation, discussions?
Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880.
### How was this PR tested?
New unit tests. `CostBasedScheduleGeneratorSpec` checks that saved-result locations reach the input port configs (alone, next to a materialized link, next to a pipelined link), that new output locations carry the context's warehouse, that the output of an operator reading a saved result is partitioned as behind a materialized link, and that reader locations keyed by an output port are refused. `ResourcePoliciesSpec` checks the same partitioning in the allocator alone. `RegionSpec` and `FriesReconfigurationAlgorithmSpec` check that a link only the operators list is not counted. All specs of the scheduling, coordinator and common packages and `scalafmtCheckAll` pass.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude (Claude Code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): schedule saved-result reads for cache reuse [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8732:
URL: https://github.com/apache/texera/pull/8732#issuecomment-5877771274
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 6 worse · ⚪ 9 noise (<±5%) · 0 without baseline
> Compared against main `bcd0ec4` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/36474737103)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 376 | 0.23 | 25,256/34,381/34,381 us | 🔴 -10.0% / 🔴 +113.7% |
| 🔴 | bs=100 sw=10 sl=64 | 798 | 0.487 | 123,120/145,377/145,377 us | 🔴 +5.2% / 🔴 +34.7% |
| ⚪ | bs=1000 sw=10 sl=64 | 914 | 0.558 | 1,085,469/1,154,904/1,154,904 us | ⚪ within ±5% / 🔴 -14.2% |
<details><summary>Baseline details</summary>
Latest main `bcd0ec4` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 376 tuples/sec | 418 tuples/sec | 807.63 tuples/sec | -10.0% | -53.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.23 MB/s | 0.255 MB/s | 0.493 MB/s | -9.8% | -53.3% |
| bs=10 sw=10 sl=64 | p50 | 25,256 us | 23,220 us | 12,578 us | +8.8% | +100.8% |
| bs=10 sw=10 sl=64 | p95 | 34,381 us | 31,334 us | 16,088 us | +9.7% | +113.7% |
| bs=10 sw=10 sl=64 | p99 | 34,381 us | 31,334 us | 19,009 us | +9.7% | +80.9% |
| bs=100 sw=10 sl=64 | throughput | 798 tuples/sec | 837 tuples/sec | 1,039 tuples/sec | -4.7% | -23.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.487 MB/s | 0.511 MB/s | 0.634 MB/s | -4.7% | -23.2% |
| bs=100 sw=10 sl=64 | p50 | 123,120 us | 117,035 us | 101,098 us | +5.2% | +21.8% |
| bs=100 sw=10 sl=64 | p95 | 145,377 us | 143,611 us | 107,890 us | +1.2% | +34.7% |
| bs=100 sw=10 sl=64 | p99 | 145,377 us | 143,611 us | 119,853 us | +1.2% | +21.3% |
| bs=1000 sw=10 sl=64 | throughput | 914 tuples/sec | 932 tuples/sec | 1,065 tuples/sec | -1.9% | -14.2% |
| bs=1000 sw=10 sl=64 | MB/s | 0.558 MB/s | 0.569 MB/s | 0.65 MB/s | -1.9% | -14.2% |
| bs=1000 sw=10 sl=64 | p50 | 1,085,469 us | 1,073,905 us | 992,940 us | +1.1% | +9.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,154,904 us | 1,117,153 us | 1,035,756 us | +3.4% | +11.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,154,904 us | 1,117,153 us | 1,062,329 us | +3.4% | +8.7% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,531.26,200,128000,376,0.230,25256.13,34381.41,34381.41
1,100,10,64,20,2504.93,2000,1280000,798,0.487,123120.22,145377.05,145377.05
2,1000,10,64,20,21884.57,20000,12800000,914,0.558,1085468.98,1154903.54,1154903.54
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): schedule saved-result reads for cache reuse [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8732:
URL: https://github.com/apache/texera/pull/8732#issuecomment-5877169191
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@Yicong-Huang`, `@mengw15`, `@Ma77Ball`
You can notify them by mentioning `@Yicong-Huang`, `@mengw15`, `@Ma77Ball` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): schedule saved-result reads for cache reuse [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8732:
URL: https://github.com/apache/texera/pull/8732#issuecomment-5877823185
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8732?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `87.50000%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 92.72%. Comparing base ([`bcd0ec4`](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e59b595`](https://app.codecov.io/gh/apache/texera/commit/e59b5951460f544e8862d3d3a23ac2603addbaa9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/texera/pull/8732?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...gine/architecture/scheduling/CacheReadInputs.scala](https://app.codecov.io/gh/apache/texera/pull/8732?src=pr&el=tree&filepath=amber%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Famber%2Fengine%2Farchitecture%2Fscheduling%2FCacheReadInputs.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YW1iZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvYW1iZXIvZW5naW5lL2FyY2hpdGVjdHVyZS9zY2hlZHVsaW5nL0NhY2hlUmVhZElucHV0cy5zY2FsYQ==) | 75.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/8732?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ne/architecture/scheduling/ScheduleGenerator.scala](https://app.codecov.io/gh/apache/texera/pull/8732?src=pr&el=tree&filepath=amber%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Famber%2Fengine%2Farchitecture%2Fscheduling%2FScheduleGenerator.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YW1iZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvYW1iZXIvZW5naW5lL2FyY2hpdGVjdHVyZS9zY2hlZHVsaW5nL1NjaGVkdWxlR2VuZXJhdG9yLnNjYWxh) | 75.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/8732?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8732 +/- ##
============================================
- Coverage 92.73% 92.72% -0.01%
- Complexity 4947 4948 +1
============================================
Files 1241 1242 +1
Lines 52608 52617 +9
Branches 6507 6507
============================================
+ Hits 48785 48790 +5
- Misses 2206 2209 +3
- Partials 1617 1618 +1
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.38% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.16% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.46% <87.50%> (-0.02%)` | :arrow_down: | |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.41% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.37% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.53% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.55% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.73% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.50% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8732/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.09% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8732?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): schedule saved-result reads for cache reuse [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on PR #8732:
URL: https://github.com/apache/texera/pull/8732#issuecomment-5896275868
@parshimers @sshiv012 could you review this one too? It stands alone; #8733 builds on it and on #8731. Design summary: https://github.com/apache/texera/discussions/5880#discussioncomment-18646157. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org