You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/09/01 23:22:29 UTC
[PR] feat(py2udf): analyze control scopes [texera]
carloea2 opened a new pull request, #8324:
URL: https://github.com/apache/texera/pull/8324
### What changes were proposed in this PR?
Add lexical-scope indexing and structured control-flow traversal for complete Python statements. The analysis derives deterministic control routes for branching, loops, exceptions, context managers, and nested lexical scopes while keeping rendering concerns outside the analysis layer.
This draft is stacked on #8323. Review this PR's control/scope commit after the preceding drafts; its branch will be rebased as the stack lands.
### Any related issues, documentation, discussions?
Related to #8319, #8322, #8323, and discussion #8160.
### How was this PR tested?
```powershell
python -m pytest py2udf/src/test/python/python_to_workflow/mosaic/test_controlwalk.py -q
python -m ruff check py2udf/src/main/python/python_to_workflow/mosaic/analysis py2udf/src/test/python/python_to_workflow/mosaic/test_controlwalk.py
python -m ruff format --check py2udf/src/main/python/python_to_workflow/mosaic/analysis py2udf/src/test/python/python_to_workflow/mosaic/test_controlwalk.py
```
Result: 2 tests passed; Ruff checks passed.
The parameterized cases cover nested branching, loops, exceptions, context managers, abrupt completion, and lexical-scope boundaries.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): analyze control scopes [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8324:
URL: https://github.com/apache/texera/pull/8324#issuecomment-5502042923
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 9 worse · ⚪ 6 noise (<±5%) · 0 without baseline
> Compared against main `fc7cfcd` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/33571333800)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 370 | 0.226 | 25,950/39,564/39,564 us | 🔴 +16.4% / 🔴 +148.8% |
| 🔴 | bs=100 sw=10 sl=64 | 796 | 0.486 | 127,423/144,069/144,069 us | 🔴 +8.0% / 🔴 +30.2% |
| 🔴 | bs=1000 sw=10 sl=64 | 909 | 0.555 | 1,096,993/1,182,492/1,182,492 us | 🔴 +5.8% / 🔴 +12.2% |
<details><summary>Baseline details</summary>
Latest main `fc7cfcd` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 370 tuples/sec | 383 tuples/sec | 750.65 tuples/sec | -3.4% | -50.7% |
| bs=10 sw=10 sl=64 | MB/s | 0.226 MB/s | 0.234 MB/s | 0.458 MB/s | -3.4% | -50.7% |
| bs=10 sw=10 sl=64 | p50 | 25,950 us | 27,212 us | 13,049 us | -4.6% | +98.9% |
| bs=10 sw=10 sl=64 | p95 | 39,564 us | 34,004 us | 15,902 us | +16.4% | +148.8% |
| bs=10 sw=10 sl=64 | p99 | 39,564 us | 34,004 us | 19,184 us | +16.4% | +106.2% |
| bs=100 sw=10 sl=64 | throughput | 796 tuples/sec | 844 tuples/sec | 960.3 tuples/sec | -5.7% | -17.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.486 MB/s | 0.515 MB/s | 0.586 MB/s | -5.6% | -17.1% |
| bs=100 sw=10 sl=64 | p50 | 127,423 us | 117,977 us | 104,199 us | +8.0% | +22.3% |
| bs=100 sw=10 sl=64 | p95 | 144,069 us | 136,693 us | 110,645 us | +5.4% | +30.2% |
| bs=100 sw=10 sl=64 | p99 | 144,069 us | 136,693 us | 120,888 us | +5.4% | +19.2% |
| bs=1000 sw=10 sl=64 | throughput | 909 tuples/sec | 927 tuples/sec | 993.27 tuples/sec | -1.9% | -8.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.555 MB/s | 0.566 MB/s | 0.606 MB/s | -1.9% | -8.5% |
| bs=1000 sw=10 sl=64 | p50 | 1,096,993 us | 1,078,784 us | 1,012,191 us | +1.7% | +8.4% |
| bs=1000 sw=10 sl=64 | p95 | 1,182,492 us | 1,117,641 us | 1,054,057 us | +5.8% | +12.2% |
| bs=1000 sw=10 sl=64 | p99 | 1,182,492 us | 1,117,641 us | 1,081,313 us | +5.8% | +9.4% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,541.01,200,128000,370,0.226,25949.58,39564.20,39564.20
1,100,10,64,20,2512.41,2000,1280000,796,0.486,127423.05,144069.11,144069.11
2,1000,10,64,20,22006.42,20000,12800000,909,0.555,1096993.48,1182491.60,1182491.60
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): analyze control scopes [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 closed pull request #8324: feat(py2udf): analyze control scopes
URL: https://github.com/apache/texera/pull/8324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): analyze control scopes [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8324:
URL: https://github.com/apache/texera/pull/8324#issuecomment-5501989608
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8324?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 94.07%. Comparing base ([`fc7cfcd`](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`880bfc3`](https://app.codecov.io/gh/apache/texera/commit/880bfc30243665cb520839888d7fa6f05a3bedcc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8324 +/- ##
=========================================
Coverage 94.07% 94.07%
Complexity 4814 4814
=========================================
Files 1196 1196
Lines 48713 48713
Branches 5866 5866
=========================================
Hits 45825 45825
Misses 1436 1436
Partials 1452 1452
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.32% <ø> (ø)` | | |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.12% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.67% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.00% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.91% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.80% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.31% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.18% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8324/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.19% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8324?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): analyze control scopes [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8324:
URL: https://github.com/apache/texera/pull/8324#issuecomment-5501881019
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`, `@Yicong-Huang`, `@Ma77Ball`
You can notify them by mentioning `@aglinxinyuan`, `@Yicong-Huang`, `@Ma77Ball` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org