You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "kz930 (via GitHub)" <gi...@apache.org> on 2026/09/01 23:37:49 UTC
[PR] test(workflow-compiling-service): run an operator both ways and compare the files [texera]
kz930 opened a new pull request, #8327:
URL: https://github.com/apache/texera/pull/8327
### What changes were proposed in this PR?
The standalone export added in #8326 claims that a generated script does what the operator does. Nothing checks it. This adds the two runners that make the claim checkable, and the file format they meet in.
`OpExecHarness` runs a `LogicalOp` the way the engine does, compiling it to a physical plan and driving the executor, but outside a workflow: its inputs are JSONL files rather than a live upstream, and its outputs are written back as files. `PyOpExecHarness` does the same for a Python operator, through the worker the engine itself uses. `StandaloneRunner` takes the other path: it asks the operator for its standalone code, wraps it in a script that binds `in1df` from the same files, runs it, and keeps the script where it ran so a failing operator can be opened as generated rather than described second-hand.
`TupleIO` is the format the two meet in. A JSONL row carries values and no types, so the schema travels beside it in a sidecar; without one, a column written as INTEGER reads back as a plain number and the two paths appear to disagree over a difference neither operator made.
Both runners produce files rather than verdicts. What to make of a difference between two files — which columns may be reordered, which values are close enough — is a separate concern and follows in a later change.
This PR sits on top of #8326, which adds the `StandaloneCodeGenerator` trait the standalone runner reads. A cross-fork pull request cannot take another fork's branch as its base, so the diff below carries #8326's commit as well. The commit to review here is the second one, `test(workflow-compiling-service): run an operator both ways and compare the files`; the diff shrinks to it once #8326 merges.
### Any related issues, documentation, discussions?
Part of #8325. Builds on #8326.
### How was this PR tested?
`HarnessSpec` runs one operator through both paths and reads the files back. Distinct is the operator throughout, because what is under test is the harness rather than the operator: it takes one input, needs no configuration, and its answer on four rows with one repeat is short enough to state in full.
The three cases are the round trip through `TupleIO`, including the sidecar schema; `OpExecHarness` producing one file per output port with the duplicate gone; and `StandaloneRunner` reaching the same three rows through the generated script.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5546784307
@aglinxinyuan May you take a look at this PR? This is the first PR of my workflow to Python project. After this first PR being merge, I can raise more PR, since this is the foundation for most of the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5518472008
@carloea2 May you take a look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5518469745
/request-review @mengw15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-compiling-service): run an operator both ways and compare the files [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5502129532
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 6 worse · ⚪ 9 noise (<±5%) · 0 without baseline
> Compared against main `fc7cfcd` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/33571875669)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 405 | 0.247 | 24,916/37,766/37,766 us | 🔴 +49.6% / 🔴 +137.5% |
| 🔴 | bs=100 sw=10 sl=64 | 893 | 0.545 | 110,168/146,384/146,384 us | 🔴 +5.6% / 🔴 +32.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,086 | 0.663 | 921,375/1,027,692/1,027,692 us | ⚪ within ±5% / 🟢 +9.4% |
<details><summary>Baseline details</summary>
Latest main `fc7cfcd` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 405 tuples/sec | 496 tuples/sec | 750.65 tuples/sec | -18.3% | -46.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.247 MB/s | 0.303 MB/s | 0.458 MB/s | -18.5% | -46.1% |
| bs=10 sw=10 sl=64 | p50 | 24,916 us | 19,243 us | 13,049 us | +29.5% | +90.9% |
| bs=10 sw=10 sl=64 | p95 | 37,766 us | 25,237 us | 15,902 us | +49.6% | +137.5% |
| bs=10 sw=10 sl=64 | p99 | 37,766 us | 25,237 us | 19,184 us | +49.6% | +96.9% |
| bs=100 sw=10 sl=64 | throughput | 893 tuples/sec | 928 tuples/sec | 960.3 tuples/sec | -3.8% | -7.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.545 MB/s | 0.567 MB/s | 0.586 MB/s | -3.9% | -7.0% |
| bs=100 sw=10 sl=64 | p50 | 110,168 us | 104,323 us | 104,199 us | +5.6% | +5.7% |
| bs=100 sw=10 sl=64 | p95 | 146,384 us | 143,927 us | 110,645 us | +1.7% | +32.3% |
| bs=100 sw=10 sl=64 | p99 | 146,384 us | 143,927 us | 120,888 us | +1.7% | +21.1% |
| bs=1000 sw=10 sl=64 | throughput | 1,086 tuples/sec | 1,076 tuples/sec | 993.27 tuples/sec | +0.9% | +9.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.663 MB/s | 0.657 MB/s | 0.606 MB/s | +0.9% | +9.4% |
| bs=1000 sw=10 sl=64 | p50 | 921,375 us | 928,209 us | 1,012,191 us | -0.7% | -9.0% |
| bs=1000 sw=10 sl=64 | p95 | 1,027,692 us | 995,288 us | 1,054,057 us | +3.3% | -2.5% |
| bs=1000 sw=10 sl=64 | p99 | 1,027,692 us | 995,288 us | 1,081,313 us | +3.3% | -5.0% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,494.13,200,128000,405,0.247,24916.49,37766.39,37766.39
1,100,10,64,20,2239.56,2000,1280000,893,0.545,110168.00,146384.23,146384.23
2,1000,10,64,20,18417.39,20000,12800000,1086,0.663,921374.99,1027692.15,1027692.15
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-compiling-service): run an operator both ways and compare the files [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5502090077
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8327?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `71.15385%` with `15 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 94.05%. Comparing base ([`8d422b7`](https://app.codecov.io/gh/apache/texera/commit/8d422b7b06b83fb14ac8078e62bf49dfedcd3444?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a7558eb`](https://app.codecov.io/gh/apache/texera/commit/a7558ebdd67973d1042f1f0d34c3a73f8237d85b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/texera/pull/8327?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ra/service/resource/WorkflowToPythonResource.scala](https://app.codecov.io/gh/apache/texera/pull/8327?src=pr&el=tree&filepath=workflow-compiling-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Fservice%2Fresource%2FWorkflowToPythonResource.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-d29ya2Zsb3ctY29tcGlsaW5nLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvc2VydmljZS9yZXNvdXJjZS9Xb3JrZmxvd1RvUHl0aG9uUmVzb3VyY2Uuc2NhbGE=) | 0.00% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/texera/pull/8327?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../amber/translator/WorkflowToPythonTranslator.scala](https://app.codecov.io/gh/apache/texera/pull/8327?src=pr&el=tree&filepath=workflow-compiling-service%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Ftexera%2Famber%2Ftranslator%2FWorkflowToPythonTranslator.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-d29ya2Zsb3ctY29tcGlsaW5nLXNlcnZpY2Uvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS90ZXhlcmEvYW1iZXIvdHJhbnNsYXRvci9Xb3JrZmxvd1RvUHl0aG9uVHJhbnNsYXRvci5zY2FsYQ==) | 85.71% | [1 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/texera/pull/8327?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8327 +/- ##
============================================
- Coverage 94.08% 94.05% -0.03%
Complexity 4812 4812
============================================
Files 1194 1196 +2
Lines 48597 48644 +47
Branches 5857 5865 +8
============================================
+ Hits 45721 45753 +32
- Misses 1427 1437 +10
- Partials 1449 1454 +5
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.32% <ø> (ø)` | | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.12% <ø> (-0.01%)` | :arrow_down: | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.67% <ø> (ø)` | | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.00% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.91% <ø> (ø)` | | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.81% <ø> (ø)` | | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.31% <ø> (ø)` | | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.18% <ø> (ø)` | | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8327/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.31% <71.15%> (-2.89%)` | :arrow_down: | Carriedforward from [5b8dd57](https://app.codecov.io/gh/apache/texera/commit/5b8dd572fd77bbae9b00088dc316dac0aaceea86?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8327?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5624983542
/request-review @aglinxinyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5624980426
/unrequest-review @mengw15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-compiling-service): run an operator both ways and compare the files [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5502059735
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`, `@Yicong-Huang`, `@mengw15`
You can notify them by mentioning `@aglinxinyuan`, `@Yicong-Huang`, `@mengw15` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8327:
URL: https://github.com/apache/texera/pull/8327#discussion_r3936147015
##########
common/pybuilder/src/main/scala/org/apache/texera/amber/pybuilder/PythonTemplateBuilder.scala:
##########
@@ -209,6 +209,29 @@ object PythonTemplateBuilder {
def wrapWithPythonDecoderExpr(text: String): String =
s"self.decode_python_template('$text')"
+ /**
+ * Render `text` as a Python double-quoted string literal, quotes included.
+ *
+ * For generators that emit standalone Python source rather than an operator
+ * for the runtime: they cannot use the decode expression (it needs the
+ * operator's `decode_python_template`, and it is deliberately rejected inside
+ * quotes), so they need the value as a *literal*. Writing `"$value"` by hand
+ * instead lets any quote, backslash or newline in the value close the literal
+ * early and change — or break — the emitted program.
+ *
+ * Escapes exactly what can end a double-quoted single-line literal.
+ */
+ def pyStringLiteral(text: String): String = {
+ val escaped = Option(text)
+ .getOrElse("")
+ .replace("\\", "\\\\")
+ .replace("\"", "\\\"")
+ .replace("\r", "\\r")
+ .replace("\n", "\\n")
+ .replace("\t", "\\t")
Review Comment:
A JSON string may contain `\u0000`, but this renderer leaves that character verbatim. Python rejects generated source containing a null byte (`SyntaxError: source code string cannot contain null bytes`), so a valid column name or predicate value can make the export unusable. Escape NUL as a Python hex escape.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/projection/ProjectionOpDesc.scala:
##########
@@ -98,4 +104,31 @@ class ProjectionOpDesc extends MapOpDesc {
outputPorts = List(OutputPort())
)
}
+
+ override def generateStandaloneCode(): String = {
+ val units = Option(attributes).getOrElse(List.empty)
+ // JVM validates non-empty at runtime via Preconditions; emit passthrough
+ // as best-effort so the standalone script still runs.
+ if (units.isEmpty) return "out1df = in1df.copy()"
Review Comment:
This silently changes an invalid Projection into a successful pass-through. The engine explicitly rejects an empty attribute list in schema propagation and `ProjectionOpExec`, so the exported workflow can produce data where the Texera workflow fails. Emit an explicit failure instead of copying the input.
This issue also appears in the following locations of the same file:
- line 114
- line 129
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala:
##########
@@ -60,4 +62,40 @@ class SpecializedFilterOpDesc extends FilterOpDesc {
supportReconfiguration = true
)
}
+
+ override def generateStandaloneCode(): String = {
+ if (predicates.isEmpty) return "out1df = in1df.copy()"
Review Comment:
An empty predicate list does not mean pass-through in the engine: `SpecializedFilterOpExec` uses `desc.predicates.exists(...)`, which is false for every tuple. This export therefore returns every row where Texera returns none. Emit an empty frame with the same columns instead.
This issue also appears on line 80 of the same file.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/limit/LimitOpDesc.scala:
##########
@@ -80,4 +80,8 @@ class LimitOpDesc extends LogicalOp {
}
Success(newPhysicalOp, Some(stateTransferFunc))
}
+
+ override def generateStandaloneCode(): String = {
+ s"out1df = in1df.head($limit).reset_index(drop=True)"
Review Comment:
Negative limits diverge from the engine: `LimitOpExec` emits zero rows because `count < limit` is immediately false, while pandas `head(-1)` emits every row except the last. Clamp the generated argument at zero (or reject negative limits consistently).
##########
workflow-compiling-service/src/main/scala/org/apache/texera/amber/translator/WorkflowToPythonTranslator.scala:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.translator
+
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.core.virtualidentity.OperatorIdentity
+import org.apache.texera.common.compiler.model.LogicalPlan
+import org.apache.texera.amber.operator.StandaloneCodeGenerator
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+import scala.jdk.CollectionConverters._
+
+class WorkflowToPythonTranslator extends LazyLogging {
+
+ // Output-port-level key. An operator with N output ports gets N entries
+ // (e.g. Split has port 0 and port 1, each with its own assigned dfN var).
+ private type PortKey = (String, Int) // (opId, portIdx)
+
+ def translate(logicalPlan: LogicalPlan): String = {
+ // Track downstream connections per (opId, fromPortIdx). A port is a leaf
+ // if it has no outgoing edges — operator-level "no outgoing links" is too
+ // coarse for multi-output ops (Split's port 0 may have downstream while
+ // port 1 doesn't, or vice versa).
+ val outgoingFromPort = mutable.Map[PortKey, Int]().withDefaultValue(0)
+ logicalPlan.links.foreach { link =>
+ outgoingFromPort((link.fromOpId.id, link.fromPortId.id)) += 1
+ }
+
+ val outputVar = mutable.Map[PortKey, String]()
+ var varCounter = 1
+ val script = ArrayBuffer[String]()
+
+ script += "import pandas as pd"
+ script += "import plotly.express as px"
+ script += "import plotly.graph_objects as go"
+ script += "import plotly.io"
Review Comment:
These unconditional imports make every exported workflow require Plotly even though all generators introduced here use only pandas. A Distinct/Filter/Limit/Projection/Union script fails at startup with `ModuleNotFoundError` in an otherwise sufficient pandas environment. Collect imports from the participating generators, as is done for helpers, or omit Plotly until an operator needs it.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala:
##########
@@ -459,6 +459,16 @@ abstract class LogicalOp extends PortDescriptor with Serializable {
def operatorInfo: OperatorInfo
+ /**
+ * Whether the row ORDER of this operator's output is part of its contract.
+ * Defaults to false: the engine runs operators across parallel workers, so
+ * for almost every operator the output row order is an implementation-defined
+ * interleaving that consumers must not rely on. Only operators whose very
+ * purpose is to establish an order (the sort family) override this to true.
+ * Consumers that must not rely on a stable row order read this flag.
+ */
+ def orderSensitive: Boolean = false
Review Comment:
This documentation says the sort family overrides the flag and consumers read it, but repository-wide usage currently finds neither an override nor a consumer. The flag therefore has no effect and is unrelated to this PR's export path; it appears to be parity-test scaffolding from a later issue in #8325. Remove it from this PR or include the behavior that makes the contract true.
##########
workflow-compiling-service/src/main/scala/org/apache/texera/amber/translator/WorkflowToPythonTranslator.scala:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.translator
+
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.core.virtualidentity.OperatorIdentity
+import org.apache.texera.common.compiler.model.LogicalPlan
+import org.apache.texera.amber.operator.StandaloneCodeGenerator
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+import scala.jdk.CollectionConverters._
+
+class WorkflowToPythonTranslator extends LazyLogging {
+
+ // Output-port-level key. An operator with N output ports gets N entries
+ // (e.g. Split has port 0 and port 1, each with its own assigned dfN var).
+ private type PortKey = (String, Int) // (opId, portIdx)
+
+ def translate(logicalPlan: LogicalPlan): String = {
+ // Track downstream connections per (opId, fromPortIdx). A port is a leaf
+ // if it has no outgoing edges — operator-level "no outgoing links" is too
+ // coarse for multi-output ops (Split's port 0 may have downstream while
+ // port 1 doesn't, or vice versa).
+ val outgoingFromPort = mutable.Map[PortKey, Int]().withDefaultValue(0)
+ logicalPlan.links.foreach { link =>
+ outgoingFromPort((link.fromOpId.id, link.fromPortId.id)) += 1
+ }
+
+ val outputVar = mutable.Map[PortKey, String]()
+ var varCounter = 1
+ val script = ArrayBuffer[String]()
+
+ script += "import pandas as pd"
+ script += "import plotly.express as px"
+ script += "import plotly.graph_objects as go"
+ script += "import plotly.io"
+ script += ""
+
+ // getTopologicalOpIds() uses jgrapht internally — no need for a custom topo sort
+ val topoOrder = logicalPlan.getTopologicalOpIds.asScala.toList
+
+ // Helper definitions the operator bodies below refer to. Collected across the
+ // whole plan and deduplicated by text, so a workflow holding two operators
+ // that share one helper still emits it once. Order follows the topological
+ // order, which keeps the script stable for a given plan.
+ val helpers = topoOrder
+ .map(logicalPlan.getOperator)
+ .collect { case gen: StandaloneCodeGenerator => gen.standaloneHelpers() }
+ .flatten
+ .distinct
+ if (helpers.nonEmpty) {
+ helpers.foreach { helper => script += helper; script += "" }
+ }
+
+ for (opIdentity <- topoOrder) {
+ val opId = opIdentity.id
+ val op = logicalPlan.getOperator(opIdentity)
+ val displayName = op.operatorInfo.userFriendlyName
+
+ // Resolve upstream inputs in the consuming operator's input-port order
+ // (link.toPortId), NOT the order links happen to appear in the plan's
+ // link list. This makes in1df/in2df/... deterministic and correct for
+ // multi-input operators (joins, set ops) where port 0 vs port 1 carries
+ // semantics (e.g. build vs probe side). Ties on the same toPortId keep
+ // link order — relevant for variadic single-port operators like Union.
+ // Each upstream link is resolved via (fromOpId, fromPortId) so that a
+ // multi-output upstream (Split) hands each downstream the correct DF.
+ val inVars = logicalPlan
+ .getUpstreamLinks(opIdentity)
+ .sortBy(link => (link.toPortId.id, link.toPortId.internal))
+ .map(link => outputVar((link.fromOpId.id, link.fromPortId.id)))
+
+ // Allocate one dfN per declared output port. Existing single-output
+ // operators have outputPorts.size == 1, so they get exactly one var and
+ // their behavior is identical to the previous flat scheme.
+ val outVars = op.operatorInfo.outputPorts.map { port =>
+ val v = s"df$varCounter"
+ varCounter += 1
+ outputVar((opId, port.id.id)) = v
+ v
+ }
+
+ script += s"# [$displayName]"
+
+ // Jackson deserializes each operator into its concrete subclass via @JsonSubTypes on LogicalOp,
+ // so the pattern match below will resolve to the correct descriptor (e.g. BarChartOpDesc).
+ op match {
+ case gen: StandaloneCodeGenerator =>
+ // generateStandaloneCode() returns a code block using in{N}df / out{N}df
+ // placeholders; substituteVars() replaces them with the assigned vars.
+ script += substituteVars(gen.generateStandaloneCode(), inVars, outVars, displayName)
+
+ case _ =>
+ logger.warn(
+ s"Operator '$displayName' does not implement StandaloneCodeGenerator. Skipping."
+ )
+ script += s"# TODO: '$displayName' is not yet supported by the translator."
+ outVars.zipWithIndex.foreach {
+ case (v, i) => script += s"# $v = <output port $i of $displayName>"
+ }
+ }
+
+ script += ""
+ }
+
+ // Leaf detection runs at the port level: a (opId, port) pair is a leaf
+ // if no link consumes it. For Split with one downstream port and one
+ // dangling port, only the dangling port is treated as a leaf to print.
+ val leafPorts = outputVar.keys.toList
+ .sortBy { case (_, portIdx) => portIdx }
+ .filter(key => outgoingFromPort(key) == 0)
+ val dataFrameLeafPorts = leafPorts.filter {
+ case (opId, _) =>
+ logicalPlan.getOperator(OperatorIdentity(opId)) match {
+ case gen: StandaloneCodeGenerator => gen.producesDataFrame()
+ case _ => false
+ }
+ }
+
+ if (dataFrameLeafPorts.nonEmpty) {
+ script += "# --- Output ---"
+ // Print in topological order of the producing operator so multi-port
+ // operators print contiguously and the order matches the script flow.
+ val topoIndex = topoOrder.map(_.id).zipWithIndex.toMap
+ dataFrameLeafPorts
+ .sortBy { case (opId, portIdx) => (topoIndex.getOrElse(opId, Int.MaxValue), portIdx) }
+ .foreach {
+ case (opId, portIdx) =>
+ val varName = outputVar((opId, portIdx))
+ val displayName =
+ logicalPlan.getOperator(OperatorIdentity(opId)).operatorInfo.userFriendlyName
+ val portSuffix = if (outputVar.keys.count(_._1 == opId) > 1) s" port $portIdx" else ""
+ script += s"""print("\\n[$displayName$portSuffix] $varName:")"""
+ script += s"print($varName.head())"
Review Comment:
This prints only the first five rows of each leaf DataFrame, although the endpoint contract says the standalone script prints its results. Any workflow producing more than five rows therefore has truncated observable output; print the full leaf DataFrame instead.
##########
workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowToPythonResource.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.service.resource
+
+import com.fasterxml.jackson.annotation.{JsonSubTypes, JsonTypeInfo}
+import com.typesafe.scalalogging.LazyLogging
+import jakarta.annotation.security.RolesAllowed
+import jakarta.ws.rs.core.MediaType
+import jakarta.ws.rs.{Consumes, POST, Path, Produces}
+import org.apache.texera.common.compiler.model.{LogicalPlan, LogicalPlanPojo}
+import org.apache.texera.amber.translator.WorkflowToPythonTranslator
+
+@JsonTypeInfo(
+ use = JsonTypeInfo.Id.NAME,
+ include = JsonTypeInfo.As.PROPERTY,
+ property = "type"
+)
+@JsonSubTypes(
+ Array(
+ new JsonSubTypes.Type(value = classOf[WorkflowToPythonSuccess], name = "success"),
+ new JsonSubTypes.Type(value = classOf[WorkflowToPythonFailure], name = "failure")
+ )
+)
+sealed trait WorkflowToPythonResponse
+
+case class WorkflowToPythonSuccess(pythonCode: String) extends WorkflowToPythonResponse
+
+case class WorkflowToPythonFailure(errorMessage: String) extends WorkflowToPythonResponse
+
+@Consumes(Array(MediaType.APPLICATION_JSON))
+@Produces(Array(MediaType.APPLICATION_JSON))
+@RolesAllowed(Array("REGULAR", "ADMIN"))
+@Path("/workflow-to-python")
+class WorkflowToPythonResource extends LazyLogging {
+
+ private val translator = new WorkflowToPythonTranslator()
+
+ @POST
+ @Path("")
+ def convertWorkflowToPython(
+ logicalPlanPojo: LogicalPlanPojo
+ ): WorkflowToPythonResponse = {
Review Comment:
The new HTTP contract has no resource-level coverage: unlike `WorkflowCompilationResourceSpec`, nothing posts frontend-shaped JSON and verifies the success/failure discriminator and generated payload, and `WorkflowCompilingServiceRunSpec` still verifies only the pre-existing resources. Add positive and malformed-plan endpoint tests plus registration/access-control assertions so removal or serialization regressions cannot pass CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #8327:
URL: https://github.com/apache/texera/pull/8327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on code in PR #8327:
URL: https://github.com/apache/texera/pull/8327#discussion_r3973442599
##########
workflow-compiling-service/src/main/scala/org/apache/texera/amber/translator/WorkflowToPythonTranslator.scala:
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.translator
+
+import com.typesafe.scalalogging.LazyLogging
+import org.apache.texera.amber.core.virtualidentity.OperatorIdentity
+import org.apache.texera.common.compiler.model.LogicalPlan
+import org.apache.texera.amber.operator.StandaloneCodeGenerator
+
+import scala.collection.mutable
+import scala.collection.mutable.ArrayBuffer
+import scala.jdk.CollectionConverters._
+
+class WorkflowToPythonTranslator extends LazyLogging {
+
+ // Output-port-level key. An operator with N output ports gets N entries
+ // (e.g. Split has port 0 and port 1, each with its own assigned dfN var).
+ private type PortKey = (String, Int) // (opId, portIdx)
+
+ def translate(logicalPlan: LogicalPlan): String = {
+ // Track downstream connections per (opId, fromPortIdx). A port is a leaf
+ // if it has no outgoing edges — operator-level "no outgoing links" is too
+ // coarse for multi-output ops (Split's port 0 may have downstream while
+ // port 1 doesn't, or vice versa).
+ val outgoingFromPort = mutable.Map[PortKey, Int]().withDefaultValue(0)
+ logicalPlan.links.foreach { link =>
+ outgoingFromPort((link.fromOpId.id, link.fromPortId.id)) += 1
+ }
+
+ val outputVar = mutable.Map[PortKey, String]()
+ var varCounter = 1
+ val script = ArrayBuffer[String]()
+
+ // getTopologicalOpIds() uses jgrapht internally — no need for a custom topo sort
+ val topoOrder = logicalPlan.getTopologicalOpIds.asScala.toList
+
+ // pandas is the one module every generator uses: an operator body reads and
+ // writes frames whatever else it does. Everything beyond that is asked of the
+ // operators in the plan, so a script that draws nothing does not require a
+ // plotting library to start.
+ script += "import pandas as pd"
+ topoOrder
+ .map(logicalPlan.getOperator)
+ .collect { case gen: StandaloneCodeGenerator => gen.standaloneImports() }
+ .flatten
+ .distinct
+ .foreach(script += _)
+ script += ""
+
+ // Helper definitions the operator bodies below refer to. Collected across the
+ // whole plan and deduplicated by text, so a workflow holding two operators
+ // that share one helper still emits it once. Order follows the topological
+ // order, which keeps the script stable for a given plan.
+ val helpers = topoOrder
+ .map(logicalPlan.getOperator)
+ .collect { case gen: StandaloneCodeGenerator => gen.standaloneHelpers() }
+ .flatten
+ .distinct
+ if (helpers.nonEmpty) {
+ helpers.foreach { helper => script += helper; script += "" }
+ }
+
+ for (opIdentity <- topoOrder) {
+ val opId = opIdentity.id
+ val op = logicalPlan.getOperator(opIdentity)
+ val displayName = op.operatorInfo.userFriendlyName
+
+ // Resolve upstream inputs in the consuming operator's input-port order
+ // (link.toPortId), NOT the order links happen to appear in the plan's
+ // link list. This makes in1df/in2df/... deterministic and correct for
+ // multi-input operators (joins, set ops) where port 0 vs port 1 carries
+ // semantics (e.g. build vs probe side). Ties on the same toPortId keep
+ // link order — relevant for variadic single-port operators like Union.
+ // Each upstream link is resolved via (fromOpId, fromPortId) so that a
+ // multi-output upstream (Split) hands each downstream the correct DF.
+ val inVars = logicalPlan
+ .getUpstreamLinks(opIdentity)
+ .sortBy(link => (link.toPortId.id, link.toPortId.internal))
+ .map(link => outputVar((link.fromOpId.id, link.fromPortId.id)))
+
+ // Allocate one dfN per declared output port. Existing single-output
+ // operators have outputPorts.size == 1, so they get exactly one var and
+ // their behavior is identical to the previous flat scheme.
+ val outVars = op.operatorInfo.outputPorts.map { port =>
+ val v = s"df$varCounter"
+ varCounter += 1
+ outputVar((opId, port.id.id)) = v
+ v
+ }
+
+ script += s"# [$displayName]"
+
+ // Jackson deserializes each operator into its concrete subclass via @JsonSubTypes on LogicalOp,
+ // so the pattern match below will resolve to the correct descriptor (e.g. BarChartOpDesc).
+ op match {
+ case gen: StandaloneCodeGenerator =>
+ // generateStandaloneCode() returns a code block using in{N}df / out{N}df
+ // placeholders; substituteVars() replaces them with the assigned vars.
+ script += substituteVars(gen.generateStandaloneCode(), inVars, outVars, displayName)
+
+ case _ =>
+ logger.warn(
+ s"Operator '$displayName' does not implement StandaloneCodeGenerator. Skipping."
+ )
+ script += s"# TODO: '$displayName' is not yet supported by the translator."
+ outVars.zipWithIndex.foreach {
+ case (v, i) => script += s"# $v = <output port $i of $displayName>"
+ }
+ }
+
+ script += ""
+ }
+
+ // Leaf detection runs at the port level: a (opId, port) pair is a leaf
+ // if no link consumes it. For Split with one downstream port and one
+ // dangling port, only the dangling port is treated as a leaf to print.
+ val leafPorts = outputVar.keys.toList
+ .sortBy { case (_, portIdx) => portIdx }
+ .filter(key => outgoingFromPort(key) == 0)
+ val dataFrameLeafPorts = leafPorts.filter {
+ case (opId, _) =>
+ logicalPlan.getOperator(OperatorIdentity(opId)) match {
+ case gen: StandaloneCodeGenerator => gen.producesDataFrame()
+ case _ => false
+ }
+ }
+
+ if (dataFrameLeafPorts.nonEmpty) {
+ script += "# --- Output ---"
+ // Print in topological order of the producing operator so multi-port
+ // operators print contiguously and the order matches the script flow.
+ val topoIndex = topoOrder.map(_.id).zipWithIndex.toMap
+ dataFrameLeafPorts
+ .sortBy { case (opId, portIdx) => (topoIndex.getOrElse(opId, Int.MaxValue), portIdx) }
+ .foreach {
+ case (opId, portIdx) =>
+ val varName = outputVar((opId, portIdx))
+ val displayName =
+ logicalPlan.getOperator(OperatorIdentity(opId)).operatorInfo.userFriendlyName
+ val portSuffix = if (outputVar.keys.count(_._1 == opId) > 1) s" port $portIdx" else ""
+ script += s"""print("\\n[$displayName$portSuffix] $varName:")"""
+ // The frame itself rather than head(): pandas already elides the
+ // middle of a long one, and it states the row and column count,
+ // which head() hides.
+ script += s"print($varName)"
+ script += ""
+ }
+ }
+
+ script.mkString("\n")
+ }
+
+ // Replaces in{N}df / out{N}df placeholders with concrete variable names.
+ // Substitutes in reverse index order to prevent partial matches (e.g. in1df
+ // inside in10df). After substitution, scans for any leftover placeholders
+ // and logs a warning — that signals a mismatch between an operator's
+ // declared port count and what its generateStandaloneCode actually emits.
+ private def substituteVars(
+ code: String,
+ inVars: List[String],
+ outVars: List[String],
+ displayName: String
+ ): String = {
+ var result = code
+
+ // A variadic port takes as many upstream links as the user draws, and an
+ // operator reading one cannot name them: `in1df`/`in2df` state a count, and
+ // whichever count it states is wrong for every other workflow. This one
+ // placeholder becomes the whole list, so the operator writes the same line
+ // whether it is fed one table or five.
+ result = result.replaceAll("""\binAlldf\b""", inVars.mkString("[", ", ", "]"))
+ inVars.zipWithIndex.reverse.foreach {
+ case (v, idx) => result = result.replaceAll(s"\\bin${idx + 1}df\\b", v)
Review Comment:
This also replaces text inside Python string literals. A Projection selecting the column `in1df` becomes `df2 = df1[["df1"]].copy()` and raises KeyError even though the input has the requested column. Please substitute identifier tokens without changing literals, and cover this through the translator rather than only testing the fragment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script, and verify the export [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5517617323
/unrequest-review @carloea2 @aglinxinyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5518307714
/request-review @carloea2 @mengw15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script, and verify the export [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5516961269
/request-review @carloea2 @aglinxinyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(workflow-compiling-service): export a workflow as a standalone Python script [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
kz930 commented on PR #8327:
URL: https://github.com/apache/texera/pull/8327#issuecomment-5629315324
Split. The five operators and the endpoint are now in #8502, and this change is the trait and the translator alone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org