You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/09/01 23:21:57 UTC
[PR] feat(py2udf): add source forest [texera]
carloea2 opened a new pull request, #8322:
URL: https://github.com/apache/texera/pull/8322
### What changes were proposed in this PR?
Add the `py2udf` Python package and its source-forest foundation. The parser builds a source-exact forest and an analysis inventory from one AST parse, preserving statement structure, source spans, parameters, imports, calls, and name access without executing user code.
The package is wired into the existing Python 3.12 CI stack with focused Ruff and pytest steps.
### Any related issues, documentation, discussions?
Related to #8319 and discussion #8160.
### How was this PR tested?
```powershell
python -m pytest py2udf/src/test/python/python_to_workflow/mosaic/test_forest.py -q
python -m ruff check py2udf/src/main/python/python_to_workflow/mosaic/forest.py py2udf/src/main/python/python_to_workflow/mosaic/source.py py2udf/src/test/python/python_to_workflow/mosaic/test_forest.py
python -m ruff format --check py2udf/src/main/python/python_to_workflow/mosaic/forest.py py2udf/src/main/python/python_to_workflow/mosaic/source.py py2udf/src/test/python/python_to_workflow/mosaic/test_forest.py
```
Result: 13 tests passed; Ruff checks passed.
The tests cover source reconstruction, nested statements, imports, calls, names, invalid Python, and parse reuse.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): add source forest [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8322:
URL: https://github.com/apache/texera/pull/8322#issuecomment-5501874211
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`, `@Yicong-Huang`, `@Ma77Ball`
You can notify them by mentioning `@aglinxinyuan`, `@Yicong-Huang`, `@Ma77Ball` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): add source forest [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8322:
URL: https://github.com/apache/texera/pull/8322#issuecomment-5501991466
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8322?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 94.07%. Comparing base ([`fc7cfcd`](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`def6662`](https://app.codecov.io/gh/apache/texera/commit/def666278645bdbad6c79577e3a82e6d3038a5af?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8322 +/- ##
=========================================
Coverage 94.07% 94.07%
Complexity 4814 4814
=========================================
Files 1196 1196
Lines 48713 48713
Branches 5866 5866
=========================================
Hits 45825 45825
Misses 1436 1436
Partials 1452 1452
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.32% <ø> (ø)` | | |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.12% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.67% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.00% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.91% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.80% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.31% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.18% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8322/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.19% <ø> (ø)` | | Carriedforward from [fc7cfcd](https://app.codecov.io/gh/apache/texera/commit/fc7cfcd9e14a68b43484a4d9ba923715076798f5?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8322?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): add source forest [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8322:
URL: https://github.com/apache/texera/pull/8322#issuecomment-5502043649
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 2 better · 🔴 6 worse · ⚪ 7 noise (<±5%) · 0 without baseline
> Compared against main `fc7cfcd` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/33571332717)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 362 | 0.221 | 25,869/39,449/39,449 us | 🔴 +13.9% / 🔴 +148.1% |
| 🔴 | bs=100 sw=10 sl=64 | 774 | 0.472 | 127,354/162,908/162,908 us | 🟢 -10.8% / 🔴 +47.2% |
| 🔴 | bs=1000 sw=10 sl=64 | 898 | 0.548 | 1,110,459/1,219,810/1,219,810 us | 🔴 +7.6% / 🔴 +15.7% |
<details><summary>Baseline details</summary>
Latest main `fc7cfcd` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 362 tuples/sec | 397 tuples/sec | 750.65 tuples/sec | -8.8% | -51.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.221 MB/s | 0.242 MB/s | 0.458 MB/s | -8.7% | -51.8% |
| bs=10 sw=10 sl=64 | p50 | 25,869 us | 22,721 us | 13,049 us | +13.9% | +98.2% |
| bs=10 sw=10 sl=64 | p95 | 39,449 us | 37,588 us | 15,902 us | +5.0% | +148.1% |
| bs=10 sw=10 sl=64 | p99 | 39,449 us | 37,588 us | 19,184 us | +5.0% | +105.6% |
| bs=100 sw=10 sl=64 | throughput | 774 tuples/sec | 801 tuples/sec | 960.3 tuples/sec | -3.4% | -19.4% |
| bs=100 sw=10 sl=64 | MB/s | 0.472 MB/s | 0.489 MB/s | 0.586 MB/s | -3.5% | -19.5% |
| bs=100 sw=10 sl=64 | p50 | 127,354 us | 118,305 us | 104,199 us | +7.6% | +22.2% |
| bs=100 sw=10 sl=64 | p95 | 162,908 us | 182,629 us | 110,645 us | -10.8% | +47.2% |
| bs=100 sw=10 sl=64 | p99 | 162,908 us | 182,629 us | 120,888 us | -10.8% | +34.8% |
| bs=1000 sw=10 sl=64 | throughput | 898 tuples/sec | 912 tuples/sec | 993.27 tuples/sec | -1.5% | -9.6% |
| bs=1000 sw=10 sl=64 | MB/s | 0.548 MB/s | 0.557 MB/s | 0.606 MB/s | -1.6% | -9.6% |
| bs=1000 sw=10 sl=64 | p50 | 1,110,459 us | 1,095,782 us | 1,012,191 us | +1.3% | +9.7% |
| bs=1000 sw=10 sl=64 | p95 | 1,219,810 us | 1,133,971 us | 1,054,057 us | +7.6% | +15.7% |
| bs=1000 sw=10 sl=64 | p99 | 1,219,810 us | 1,133,971 us | 1,081,313 us | +7.6% | +12.8% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,552.66,200,128000,362,0.221,25868.92,39449.03,39449.03
1,100,10,64,20,2585.20,2000,1280000,774,0.472,127353.60,162908.26,162908.26
2,1000,10,64,20,22259.51,20000,12800000,898,0.548,1110459.40,1219810.16,1219810.16
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(py2udf): add source forest [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 closed pull request #8322: feat(py2udf): add source forest
URL: https://github.com/apache/texera/pull/8322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org