You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "kz930 (via GitHub)" <gi...@apache.org> on 2026/09/02 19:51:43 UTC
[PR] feat(operator): export the machine-learning operators as Python [texera]
kz930 opened a new pull request, #8348:
URL: https://github.com/apache/texera/pull/8348
### What changes were proposed in this PR?
The last operator family implements `StandaloneCodeGenerator`: the scikit-learn estimators and their training counterparts, the four advanced trainers that read their hyperparameters from a table, the linear regression, the scorer, and the four Hugging Face models.
A model operator's script is longer than a transform's because the engine does more for it. The estimators are fitted on one port and scored on the other, so the script holds both frames and narrows each by the same rule. A fit and a score taken on different columns would compare two different models.
`JsonSchemaCustomizer` and its caller in `OperatorMetadataGenerator` come along because the advanced trainers implement it. The rules a hyperparameter's value follows are written into the schema at generation time, and without the call the schema carries none of them.
Sklearn Prediction and Sklearn Testing are reported as unverifiable rather than exported blind. Each consumes a fitted model on an input port, and a fixture written from the JVM cannot carry a live Python object.
### Any related issues, documentation, discussions?
Part of #8325. Builds on #8327, which adds the trait and the verification.
### How was this PR tested?
`OperatorBehaviorSpec` runs each operator twice and compares the frames, across the configurations each schema offers — 73 runs, 2 withheld for the reason above.
`WorkflowOperator` passes in full at 2583 tests.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(machine-learning): export the four advanced trainers [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 closed pull request #8348: feat(machine-learning): export the four advanced trainers
URL: https://github.com/apache/texera/pull/8348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(operator): export the machine-learning operators as Python [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8348:
URL: https://github.com/apache/texera/pull/8348#issuecomment-5515485910
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8348?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 93.91%. Comparing base ([`fe76eec`](https://app.codecov.io/gh/apache/texera/commit/fe76eec2dc01522ab41af1a915e87aabb2954ed8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`009b5e4`](https://app.codecov.io/gh/apache/texera/commit/009b5e44c7a357f93679416d1b29c310f6b4c798?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 5 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8348 +/- ##
============================================
- Coverage 94.05% 93.91% -0.15%
+ Complexity 4812 4740 -72
============================================
Files 1196 1184 -12
Lines 48749 47420 -1329
Branches 5884 5256 -628
============================================
- Hits 45852 44534 -1318
- Misses 1437 1473 +36
+ Partials 1460 1413 -47
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.32% <ø> (ø)` | | |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.97% <ø> (-0.12%)` | :arrow_down: | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.67% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.73% <ø> (-0.27%)` | :arrow_down: | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.91% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.48% <ø> (-0.32%)` | :arrow_down: | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.31% <ø> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.87% <ø> (+0.69%)` | :arrow_up: | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8348/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.19% <100.00%> (ø)` | | Carriedforward from [70c2114](https://app.codecov.io/gh/apache/texera/commit/70c21145887920528d7d5540e3fb790b43e8b759?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8348?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(operator): export the machine-learning operators as Python [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8348:
URL: https://github.com/apache/texera/pull/8348#issuecomment-5515583283
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline
> Compared against main `7976744` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/33675832240)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 358 | 0.218 | 26,951/33,300/33,300 us | 🔴 -11.0% / 🔴 +108.2% |
| 🔴 | bs=100 sw=10 sl=64 | 763 | 0.466 | 126,313/171,453/171,453 us | 🔴 +8.5% / 🔴 +54.9% |
| 🔴 | bs=1000 sw=10 sl=64 | 912 | 0.557 | 1,082,549/1,224,690/1,224,690 us | 🔴 +7.8% / 🔴 +16.0% |
<details><summary>Baseline details</summary>
Latest main `7976744` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 358 tuples/sec | 401 tuples/sec | 750.25 tuples/sec | -10.7% | -52.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.218 MB/s | 0.245 MB/s | 0.458 MB/s | -11.0% | -52.4% |
| bs=10 sw=10 sl=64 | p50 | 26,951 us | 26,454 us | 13,041 us | +1.9% | +106.7% |
| bs=10 sw=10 sl=64 | p95 | 33,300 us | 34,541 us | 15,996 us | -3.6% | +108.2% |
| bs=10 sw=10 sl=64 | p99 | 33,300 us | 34,541 us | 19,395 us | -3.6% | +71.7% |
| bs=100 sw=10 sl=64 | throughput | 763 tuples/sec | 831 tuples/sec | 961.14 tuples/sec | -8.2% | -20.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.466 MB/s | 0.507 MB/s | 0.587 MB/s | -8.1% | -20.6% |
| bs=100 sw=10 sl=64 | p50 | 126,313 us | 116,439 us | 103,974 us | +8.5% | +21.5% |
| bs=100 sw=10 sl=64 | p95 | 171,453 us | 166,390 us | 110,662 us | +3.0% | +54.9% |
| bs=100 sw=10 sl=64 | p99 | 171,453 us | 166,390 us | 120,690 us | +3.0% | +42.1% |
| bs=1000 sw=10 sl=64 | throughput | 912 tuples/sec | 940 tuples/sec | 991.91 tuples/sec | -3.0% | -8.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.557 MB/s | 0.573 MB/s | 0.605 MB/s | -2.8% | -8.0% |
| bs=1000 sw=10 sl=64 | p50 | 1,082,549 us | 1,064,795 us | 1,013,755 us | +1.7% | +6.8% |
| bs=1000 sw=10 sl=64 | p95 | 1,224,690 us | 1,136,316 us | 1,056,115 us | +7.8% | +16.0% |
| bs=1000 sw=10 sl=64 | p99 | 1,224,690 us | 1,136,316 us | 1,082,840 us | +7.8% | +13.1% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,559.22,200,128000,358,0.218,26951.21,33299.64,33299.64
1,100,10,64,20,2620.21,2000,1280000,763,0.466,126312.69,171452.87,171452.87
2,1000,10,64,20,21930.26,20000,12800000,912,0.557,1082549.33,1224690.20,1224690.20
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(operator): export the machine-learning operators as Python [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8348:
URL: https://github.com/apache/texera/pull/8348#issuecomment-5515470360
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`, `@Yicong-Huang`, `@carloea2`
You can notify them by mentioning `@aglinxinyuan`, `@Yicong-Huang`, `@carloea2` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(machine-learning): export the four advanced trainers [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #8348:
URL: https://github.com/apache/texera/pull/8348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org