You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "kz930 (via GitHub)" <gi...@apache.org> on 2026/07/28 21:12:42 UTC
[D] CI for workflow-to-Python verification: pooled Python workers [texera]
GitHub user kz930 created a discussion: CI for workflow-to-Python verification: pooled Python workers
Design of the verification itself is in #6072 — this is only about why it was slow and what fixed it.
### How we check one operator
Three steps, and each one that needs Python used to start a brand-new Python process:
1. **Run it the normal Texera way** — the ground truth. Scala operators run in the JVM, no Python needed. Python-based operators (Sort, most charts) start a Python process.
2. **Run the standalone Python script** our translator generated for it — always Python.
3. **Compare the two outputs** — a small pandas script, also Python.
So one check = **2 Python starts for a Scala operator, 3 for a Python-based one**. And we do one check per enum option (line chart mode = line / dots / line+dots), so a single operator often means several checks.
### The problem
- Starting Python + importing pandas/pyamber: **~300 ms**.
- The operator's actual work on our small test table: **~a few ms**.
- So **~96% of the time was imports** — repaid on every single run, across 148 discovered operators.
### The fix
- **Keep Python alive.** Start it once, import once, then feed it jobs: one job in on stdin, result out on stdout. Every job after the first skips the imports.
- **Batch.** Many operators share one worker, so imports are paid per worker, not per run.
- **One pool per role** — Texera path, generated script, comparison. Three different worker scripts (and the first needs its own `PYTHONPATH`), so workers are never mixed between them.
- **Run 4 at a time.** Each pool keeps up to 4 workers; a worker takes one job at a time, since it changes directory per job.
- **If a worker crashes**, we drop it and run that one job the old way — never worse than before. A normal test failure isn't a crash; the worker reports it and stays alive. `VERIFY_PYTHON_WORKER=0` turns pooling off entirely.
- **In CI**, it's a step in the existing platform-integration job for workflow-compiling-service (Python and the service are already there). The unit job skips it by tag and stays fast.
### The result
Same suite, same 56 operators, same machine:
| | test run | wall clock | CPU used |
|---|---|---|---|
| workers on | **8.0 s** | 14.2 s | 61 s |
| workers off | 32.8 s | 39.3 s | 201 s |
- **~4× faster, ~3× less CPU.**
- CI is green — 53 operators at the last run.
### How to test it
Point `UDF_PYTHON_PATH` at a Python 3.12 that has pandas, plotly, pyarrow and betterproto, then:
`UDF_PYTHON_PATH="$PWD/../venv312/bin/python" sbt "WorkflowCompilingService/testOnly *OperatorBehaviorSpec"`
GitHub link: https://github.com/apache/texera/discussions/6975
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] CI for workflow-to-Python verification: pooled Python workers [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
GitHub user kz930 edited a discussion: CI for workflow-to-Python verification: pooled Python workers
Design of the verification itself is in #6072 — this is only about why it was slow and what fixed it.
### How we check one operator
Three steps, and each one that needs Python used to start a brand-new Python process:
1. **Run it the normal Texera way** — the ground truth. Scala operators run in the JVM, no Python needed. Python-based operators (Sort, most charts) start a Python process.
2. **Run the standalone Python script** our translator generated for it — always Python.
3. **Compare the two outputs** — a small pandas script, also Python.
So one check = **2 Python starts for a Scala operator, 3 for a Python-based one**. And we do one check per enum option (line chart mode = line / dots / line+dots), so a single operator often means several checks.
### The problem
- Starting Python + importing pandas/pyamber: **~300 ms**.
- The operator's actual work on our small test table: **~a few ms**.
- So **~96% of the time was imports** — repaid on every single run, across 148 discovered operators.
### The fix
- **Keep Python alive.** Start it once, import once, then feed it jobs: one job in on stdin, result out on stdout. Every job after the first skips the imports.
- **Batch.** Many operators share one worker, so imports are paid per worker, not per run.
- **One pool per role** — Texera path, generated script, comparison. Three different worker scripts (and the first needs its own `PYTHONPATH`), so workers are never mixed between them.
- **Run 4 at a time.** Each pool keeps up to 4 workers; a worker takes one job at a time, since it changes directory per job.
- **If a worker crashes**, we drop it and run that one job the old way — never worse than before. A normal test failure isn't a crash; the worker reports it and stays alive. `VERIFY_PYTHON_WORKER=0` turns pooling off entirely.
- **In CI**, it's a step in the existing platform-integration job for workflow-compiling-service (Python and the service are already there). The unit job skips it by tag and stays fast.
### The result
Same suite, same 53 operators, same machine:
| | test run | wall clock | CPU used |
|---|---|---|---|
| workers on | **8.0 s** | 14.2 s | 61 s |
| workers off | 32.8 s | 39.3 s | 201 s |
- **~4× faster, ~3× less CPU.**
- CI is green — 53 operators at the last run.
### How to test it
Point `UDF_PYTHON_PATH` at a Python 3.12 that has pandas, plotly, pyarrow and betterproto, then:
`UDF_PYTHON_PATH="$PWD/../venv312/bin/python" sbt "WorkflowCompilingService/testOnly *OperatorBehaviorSpec"`
GitHub link: https://github.com/apache/texera/discussions/6975
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] CI for workflow-to-Python verification: pooled Python workers [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
GitHub user carloea2 added a comment to the discussion: CI for workflow-to-Python verification: pooled Python workers
It looks good to me
GitHub link: https://github.com/apache/texera/discussions/6975#discussioncomment-17958993
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] CI for workflow-to-Python verification: pooled Python workers [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
GitHub user kz930 closed a discussion: CI for workflow-to-Python verification: pooled Python workers
Design of the verification itself is in #6072 — this is only about why it was slow and what fixed it.
### How we check one operator
Three steps, and each one that needs Python used to start a brand-new Python process:
1. **Run it the normal Texera way** — the ground truth. Scala operators run in the JVM, no Python needed. Python-based operators (Sort, most charts) start a Python process.
2. **Run the standalone Python script** our translator generated for it — always Python.
3. **Compare the two outputs** — a small pandas script, also Python.
So one check = **2 Python starts for a Scala operator, 3 for a Python-based one**. And we do one check per enum option (line chart mode = line / dots / line+dots), so a single operator often means several checks.
### The problem
- Starting Python + importing pandas/pyamber: **~300 ms**.
- The operator's actual work on our small test table: **~a few ms**.
- So **~96% of the time was imports** — repaid on every single run, across 148 discovered operators.
### The fix
- **Keep Python alive.** Start it once, import once, then feed it jobs: one job in on stdin, result out on stdout. Every job after the first skips the imports.
- **Batch.** Many operators share one worker, so imports are paid per worker, not per run.
- **One pool per role** — Texera path, generated script, comparison. Three different worker scripts (and the first needs its own `PYTHONPATH`), so workers are never mixed between them.
- **Run 4 at a time.** Each pool keeps up to 4 workers; a worker takes one job at a time, since it changes directory per job.
- **If a worker crashes**, we drop it and run that one job the old way — never worse than before. A normal test failure isn't a crash; the worker reports it and stays alive. `TEXERA_TEST_PYTHON_WORKER=0` turns pooling off entirely.
- **In CI**, it's a step in the existing platform-integration job for workflow-compiling-service (Python and the service are already there). The unit job skips it by tag and stays fast.
### The result
Same suite, same 53 operators, same machine:
| | test run | wall clock | CPU used |
|---|---|---|---|
| workers on | **8.0 s** | 14.2 s | 61 s |
| workers off | 32.8 s | 39.3 s | 201 s |
- **~4× faster, ~3× less CPU.**
- CI is green — 53 operators at the last run.
### How to test it
Point `UDF_PYTHON_PATH` at a Python 3.12 that has pandas, plotly, pyarrow and betterproto, then:
`UDF_PYTHON_PATH="$PWD/../venv312/bin/python" sbt "WorkflowCompilingService/testOnly *OperatorBehaviorSpec"`
GitHub link: https://github.com/apache/texera/discussions/6975
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] CI for workflow-to-Python verification: pooled Python workers [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
GitHub user kz930 added a comment to the discussion: CI for workflow-to-Python verification: pooled Python workers
@carloea2 @Yicong-Huang
GitHub link: https://github.com/apache/texera/discussions/6975#discussioncomment-17819023
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] CI for workflow-to-Python verification: pooled Python workers [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
GitHub user kz930 edited a discussion: CI for workflow-to-Python verification: pooled Python workers
Design of the verification itself is in #6072 — this is only about why it was slow and what fixed it.
### How we check one operator
Three steps, and each one that needs Python used to start a brand-new Python process:
1. **Run it the normal Texera way** — the ground truth. Scala operators run in the JVM, no Python needed. Python-based operators (Sort, most charts) start a Python process.
2. **Run the standalone Python script** our translator generated for it — always Python.
3. **Compare the two outputs** — a small pandas script, also Python.
So one check = **2 Python starts for a Scala operator, 3 for a Python-based one**. And we do one check per enum option (line chart mode = line / dots / line+dots), so a single operator often means several checks.
### The problem
- Starting Python + importing pandas/pyamber: **~300 ms**.
- The operator's actual work on our small test table: **~a few ms**.
- So **~96% of the time was imports** — repaid on every single run, across 148 discovered operators.
### The fix
- **Keep Python alive.** Start it once, import once, then feed it jobs: one job in on stdin, result out on stdout. Every job after the first skips the imports.
- **Batch.** Many operators share one worker, so imports are paid per worker, not per run.
- **One pool per role** — Texera path, generated script, comparison. Three different worker scripts (and the first needs its own `PYTHONPATH`), so workers are never mixed between them.
- **Run 4 at a time.** Each pool keeps up to 4 workers; a worker takes one job at a time, since it changes directory per job.
- **If a worker crashes**, we drop it and run that one job the old way — never worse than before. A normal test failure isn't a crash; the worker reports it and stays alive. `TEXERA_TEST_PYTHON_WORKER=0` turns pooling off entirely.
- **In CI**, it's a step in the existing platform-integration job for workflow-compiling-service (Python and the service are already there). The unit job skips it by tag and stays fast.
### The result
Same suite, same 53 operators, same machine:
| | test run | wall clock | CPU used |
|---|---|---|---|
| workers on | **8.0 s** | 14.2 s | 61 s |
| workers off | 32.8 s | 39.3 s | 201 s |
- **~4× faster, ~3× less CPU.**
- CI is green — 53 operators at the last run.
### How to test it
Point `UDF_PYTHON_PATH` at a Python 3.12 that has pandas, plotly, pyarrow and betterproto, then:
`UDF_PYTHON_PATH="$PWD/../venv312/bin/python" sbt "WorkflowCompilingService/testOnly *OperatorBehaviorSpec"`
GitHub link: https://github.com/apache/texera/discussions/6975
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] CI for workflow-to-Python verification: pooled Python workers [texera]
Posted by "kz930 (via GitHub)" <gi...@apache.org>.
GitHub user kz930 closed the discussion with a comment: CI for workflow-to-Python verification: pooled Python workers
If it looks good, then I would close this discussion. This is the current implementation so far.
GitHub link: https://github.com/apache/texera/discussions/6975#discussioncomment-17967198
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] CI for workflow-to-Python verification: pooled Python workers [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: CI for workflow-to-Python verification: pooled Python workers
@carloea2 @Yicong-Huang : any input?
@kz930 Please lead the discussion and close it if when it's done.
GitHub link: https://github.com/apache/texera/discussions/6975#discussioncomment-17956101
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org