You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "yangzhang75 (via GitHub)" <gi...@apache.org> on 2026/06/23 22:23:50 UTC
[PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
yangzhang75 opened a new pull request, #5917:
URL: https://github.com/apache/texera/pull/5917
<!--
Thanks for sending a pull request (PR)! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
[Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md)
2. Ensure you have added or run the appropriate tests for your PR
3. If the PR is work in progress, mark it a draft on GitHub.
4. Please write your PR title to summarize what this PR proposes, we
are following Conventional Commits style for PR titles as well.
5. Be sure to keep the PR description updated to reflect all changes.
-->
### What changes were proposed in this PR?
<!--
Please clarify what changes you are proposing. The purpose of this section
is to outline the changes. Here are some tips for you:
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
3. If it is a refactoring, clarify what has been changed.
3. It would be helpful to include a before-and-after comparison using
screenshots or GIFs.
4. Please consider writing useful notes for better and faster reviews.
-->
Follow-up fix for #5597, which started passing the Python worker startup config as a
single JSON-string command-line argument. On Windows the JVM assembles argv into one
command line and the inner double quotes are stripped before Python receives them, so
`json.loads` fails with `JSONDecodeError: Expecting property name enclosed in double quotes`.
Linux/macOS are unaffected (the JVM passes argv directly, quotes survive).
This Base64-encodes the JSON on the JVM side (`encodeStartupConfig`) and Base64-decodes it
in `texera_run_python_worker.py` before parsing. Base64 uses only `[A-Za-z0-9+/=]`, so the
argument carries no quotes or spaces and survives argv quoting on every platform. The 19-key
contract and all validation are unchanged.
### Any related issues, documentation, discussions?
<!--
Please use this section to link other resources if not mentioned already.
1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves #1234`
or `Closes #1234`. If it is only related, simply mention the issue number.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
Closes #5916
### How was this PR tested?
<!--
If tests were added, say they were added here. Or simply mention that if the PR
is tested with existing test cases. Make sure to include/update test cases that
check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how
you tested step by step, ideally copy and paste-able, so that other reviewers can
test and check, and descendants can verify in the future. If tests were not added,
please describe why they were not added and/or why it was difficult to add.
-->
- Scala: `PythonWorkflowWorkerStartupConfigSpec` updated to decode Base64; added a regression
test asserting the encoded output contains no quotes/whitespace. 5/5 pass; scalafmt clean.
- Python: `test_run_python_worker.py` updated to feed Base64 input; added a round-trip test.
ruff check + format clean.
- Verified the Base64 round-trip is byte-compatible between Java `Base64.getEncoder` and
Python `base64.b64decode`.
### Was this PR authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please include the phrase: 'Generated-by: ' followed by the name of the tool
and its version. If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4784005023
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@Ma77Ball`, `@Yicong-Huang`
You can notify them by mentioning `@Ma77Ball`, `@Yicong-Huang` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4785246022
@yangzhang75 please rebase your commit history on this branch. right now it contains 4461 commits.
I marked the PR into a draft for now. please reopen it when ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4795597633
Ok I am doing testing now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang merged PR #5917:
URL: https://github.com/apache/texera/pull/5917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4803786492
as this is a fix for a refactor done in https://github.com/apache/texera/pull/5597, we don't need to backport it to v1.2
cc @xuang7 for context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4785096929
as we lack of windows dev machine and CI setup, @carloea2 could you please help test and review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "xuang7 (via GitHub)" <gi...@apache.org>.
xuang7 commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4784313979
It seems this branch has diverged from the current main branch. Could you clean up the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4795956908
It worked, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4822865983
@yangzhang75 could you please make CI work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4784381963
<!-- texera-benchmarks-comment -->
## ✅ No material benchmark regressions detected
🟢 1 better · 🔴 0 worse · ⚪ 14 noise (<±5%) · 0 without baseline
> Compared against main `c3161f7` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/28063769099)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🟢 | bs=10 sw=10 sl=64 | 420 | 0.257 | 22,930/32,564/32,564 us | 🟢 -7.1% / 🟢 -6.9% |
| ⚪ | bs=100 sw=10 sl=64 | 942 | 0.575 | 105,251/121,205/121,205 us | ⚪ within ±5% / 🟢 -13.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,125 | 0.686 | 885,763/934,120/934,120 us | ⚪ within ±5% / 🟢 -8.9% |
<details><summary>Baseline details</summary>
Latest main `c3161f7` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 420 tuples/sec | 434 tuples/sec | 410.82 tuples/sec | -3.2% | +2.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.257 MB/s | 0.265 MB/s | 0.251 MB/s | -3.0% | +2.5% |
| bs=10 sw=10 sl=64 | p50 | 22,930 us | 24,679 us | 23,785 us | -7.1% | -3.6% |
| bs=10 sw=10 sl=64 | p95 | 32,564 us | 34,274 us | 34,980 us | -5.0% | -6.9% |
| bs=10 sw=10 sl=64 | p99 | 32,564 us | 34,274 us | 34,980 us | -5.0% | -6.9% |
| bs=100 sw=10 sl=64 | throughput | 942 tuples/sec | 958 tuples/sec | 891.94 tuples/sec | -1.7% | +5.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.575 MB/s | 0.585 MB/s | 0.544 MB/s | -1.7% | +5.6% |
| bs=100 sw=10 sl=64 | p50 | 105,251 us | 104,354 us | 112,277 us | +0.9% | -6.3% |
| bs=100 sw=10 sl=64 | p95 | 121,205 us | 126,421 us | 139,802 us | -4.1% | -13.3% |
| bs=100 sw=10 sl=64 | p99 | 121,205 us | 126,421 us | 139,802 us | -4.1% | -13.3% |
| bs=1000 sw=10 sl=64 | throughput | 1,125 tuples/sec | 1,097 tuples/sec | 1,041 tuples/sec | +2.6% | +8.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.686 MB/s | 0.67 MB/s | 0.635 MB/s | +2.4% | +8.0% |
| bs=1000 sw=10 sl=64 | p50 | 885,763 us | 912,255 us | 972,714 us | -2.9% | -8.9% |
| bs=1000 sw=10 sl=64 | p95 | 934,120 us | 980,352 us | 1,023,057 us | -4.7% | -8.7% |
| bs=1000 sw=10 sl=64 | p99 | 934,120 us | 980,352 us | 1,023,057 us | -4.7% | -8.7% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,475.73,200,128000,420,0.257,22929.70,32564.37,32564.37
1,100,10,64,20,2124.01,2000,1280000,942,0.575,105250.58,121204.62,121204.62
2,1000,10,64,20,17782.12,20000,12800000,1125,0.686,885763.18,934119.85,934119.85
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4784370381
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/5917?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 54.50%. Comparing base ([`c3161f7`](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`28c8df1`](https://app.codecov.io/gh/apache/texera/commit/28c8df195dfb5121d7f6074e758810604e6b7518?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #5917 +/- ##
=========================================
Coverage 54.50% 54.50%
Complexity 2915 2915
=========================================
Files 1108 1108
Lines 42807 42807
Branches 4604 4604
=========================================
Hits 23332 23332
Misses 18119 18119
Partials 1356 1356
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.44% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.36% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.52% <ø> (-0.01%)` | :arrow_down: | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `1.65% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.35% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.59% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.27% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.20% <100.00%> (+<0.01%)` | :arrow_up: | |
| [python](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.76% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/5917/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.69% <ø> (ø)` | | Carriedforward from [c3161f7](https://app.codecov.io/gh/apache/texera/commit/c3161f79e26a37bd5ac0190603d50ec875d6548e?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/5917?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4786641523
By default, PR is ready for review. If not, author should mark it a draft
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-4786597539
Sure, feel free to ping me when is ready for a review pass. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(engine): Base64-encode Python worker startup config to survive Windows argv [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #5917:
URL: https://github.com/apache/texera/pull/5917#issuecomment-5126196656
Backport PR opened: draft#7105 (https://github.com/apache/texera/pull/7105) to `release/v1.2`, assigned to @yangzhang75 — part of a backport-coverage audit (the cherry-pick conflicts and needs manual resolution).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org