You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "sshiv012 (via GitHub)" <gi...@apache.org> on 2026/09/28 23:17:11 UTC
[PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
sshiv012 opened a new pull request, #8737:
URL: https://github.com/apache/texera/pull/8737
### What changes were proposed in this PR?
This makes Python UDF mistakes explain themselves, and rewrites the user docs for people who aren't programmers. The motivating case, from the UMich team, is in #8736.
**UDF templates** (Python UDF, 2-in, 1-out)
- `from pytexera import *` is no longer commented out, so uncommenting one class gives a working UDF.
- A short header says: keep one class, which method goes with which base class, `yield` not `return`, and `open()` / `super().__init__()`.
- The UiParameter lines are kept. Saved workflows keep their own code.
**Load errors** (`executor_manager.py`)
The old "There should be one and only one Operator defined" sentence stays first, then the message says what to fix:
| Mistake | Message now adds |
|---|---|
| No class | `No Operator class was found. Uncomment or write exactly one class...` |
| Wrong method for the base | `WrapperClass subclasses UDFOperatorV2 but does not implement process_tuple. If you meant process_table, subclass UDFTableOperator instead.` |
| Two classes | `Found 2: FirstOperator, SecondOperator. Keep one...` |
| Missing import | `name 'UDFOperatorV2' is not defined (line 2 of the UDF code). Add from pytexera import * at the top of the UDF.` |
| Any other load error | the line number in the user's code |
**Runtime errors** (`operator.py`, `data_processor.py`)
- `return` instead of `yield` now says `process_table must yield results, not return them` instead of `MatchError` / `'NoneType' object is not iterable`. Returning a list still works.
- Table and batch buffers are created lazily, so a missing `super().__init__()` no longer crashes with `'_TableOperator__table_data'`. `Operator` already does this for its template decoder.
- Yielding a dict of DataFrames, or columns that don't match the output schema, now points to the fix.
**Frontend**
- The Kill button has a tooltip.
- The empty Result panel now says: "Click an operator to see its results and Console. Click an empty spot on the canvas to see errors from all operators." Startup errors only show in the Static Error tab when nothing is selected, and users don't find that.
**Computing unit quota error** now adds: "Reuse one of your running computing units, or terminate one you no longer need."
**Docs**
- Rewritten: the Python UDF guide (a first example, the 5 rules, and a table mapping each error to its fix), the "how to use Texera" guide (colors, where errors show, a stuck-workflow checklist), and the dataset tutorial. The old dataset tutorial pointed to an `environment` tab that no longer exists, and its sample CSV link returned 404.
- New pages: Computing Units, Sharing and Permissions, and How Texera Runs Your Workflow (blocking operators, regions).
- Screenshots are from the current UI. The pages that have images became folders (`page/index.md` with the images next to it), so the URLs stay the same.
The commits are split by area if you'd rather review them one at a time.
### Any related issues, documentation, discussions?
Closes #8736
Related: #8667 / #8668 fix the silent hang on UDF setup failures. Once that lands, users will see these messages.
### How was this PR tested?
Every behavior change has a test written first.
- Python: new tests in `test_executor_manager.py`, `test_operator.py` and `test_data_processor.py`. Full suite: `cd amber && pytest -m "not integration"` gives 1349 passed. ruff check and format are clean.
- Scala: new specs for all three templates, which read the `defaultValue` and check that uncommenting any one class leaves exactly one active class. The quota message spec is updated. Checked with `WorkflowOperator/testOnly ...udf.python.*`, `ComputingUnitManagingService/testOnly ...ComputingUnitManagingServiceExceptionSpec`, and scalafmtCheck.
- Frontend: new specs in `result-panel.component.spec.ts` and `menu.component.spec.ts` (197 passed), plus prettier and eslint.
- Docs: a separate reviewer followed every tutorial in a local Texera as a first-time user. The dataset tutorial reproduced the screenshot numbers (100 → 23 → 6), and the steps it tripped on are fixed in the last commit.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
Posted by "sshiv012 (via GitHub)" <gi...@apache.org>.
sshiv012 commented on PR #8737:
URL: https://github.com/apache/texera/pull/8737#issuecomment-5904571141
Thanks @Xiao-zhen-Liu, both points make sense.
I split this into six PRs, one per area. Each targets `main` on its own:
- #8768 pyamber: load error messages
- #8769 pyamber: runtime errors and lazy table/batch buffers
- #8770 workflow-operator: Python UDF templates
- #8771 frontend: Kill tooltip and empty Result panel text
- #8772 computing-unit-managing-service: quota message
- #8773 docs, a draft until the others merge
The code PRs have before and after screenshots taken on a local Texera built from those changes. The quota one has none, since that dialog only shows on Kubernetes. In the docs PR I retook the two images you flagged (the Console error and the Static Error) with the new messages.
Closing this one in favor of those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
Posted by "sshiv012 (via GitHub)" <gi...@apache.org>.
sshiv012 closed pull request #8737: feat: explain Python UDF mistakes and rewrite user docs for non-programmers
URL: https://github.com/apache/texera/pull/8737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
Posted by "sshiv012 (via GitHub)" <gi...@apache.org>.
sshiv012 commented on code in PR #8737:
URL: https://github.com/apache/texera/pull/8737#discussion_r4140963074
##########
docs/tutorials/guide-to-use-python-udf/index.md:
##########
@@ -0,0 +1,210 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+---
+title: "Guide to Use a Python UDF"
+weight: 30
+---
+
+A **Python UDF** (user-defined function) is an operator that runs a small piece of Python code you write. Use it when no built-in operator does what you need.
+
+You don't need to be a programmer. Copy the example below and change one line at a time.
+
+## Which Python operator should I use?
+
+| You want to… | Use |
+|---|---|
+| Add or change a column with a one-line formula, e.g. `tuple_["Units Sold"] * 2` | **Python Lambda Function** (no class to write) |
+| Turn a whole table into one summary row, e.g. `table["Units Sold"].mean()` | **Python Table Reducer** (no class to write) |
+| Anything else with one or more inputs | **Python UDF** |
+| Apply a model (input 0) to data rows (input 1) | **2-in Python UDF** |
+| Create data from nothing (no input) | **1-out Python UDF** |
+
+## Your first UDF in 4 steps
+
+This example adds a `Revenue` column (units × price) to the sales data from [the dataset tutorial](../create-dataset-upload-data/).
+
+**1. Add the operator.** Drag **Python UDF** (under *User-defined Functions → Python*) onto the canvas and connect your data to it.
+
+**2. Write the code.** Click **Edit code content**. The editor opens with a template. Its classes are commented out (their lines start with `#`), so running it unchanged fails. Uncomment one class, or replace everything with:
+
+```python
+from pytexera import *
+
+
+class ProcessTupleOperator(UDFOperatorV2):
+
+ @overrides
+ def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
+ tuple_["Revenue"] = tuple_["Units Sold"] * tuple_["Unit Price"]
+ yield tuple_
+```
+
+
+
+**3. Declare the new column.** In the property panel, under **Extra output column(s)**, click **+** and add `Revenue` with type `double`. Leave **Retain input columns** checked to keep the original columns.
+
+<img src="udf-output-columns.png" alt="Retain input columns and one extra output column named Revenue" width="260">
+
+**4. Run.** Click the eye icon on the UDF, then **Run**. Click the UDF, then the **Result** tab in the result panel.
+
+## The 5 rules
+
+Almost every UDF problem comes from breaking one of these.
+
+| # | Rule | If you break it |
+|---|---|---|
+| 1 | The first line is `from pytexera import *` | `name 'UDFOperatorV2' is not defined` |
+| 2 | Exactly **one** class. Delete or comment out the others. | `There should be one and only one Operator defined` |
+| 3 | The method name must match the class you inherit from (table below) | `... No super class method found`, or rule 2's error |
+| 4 | Send results with `yield`, never `return` | ``process_tuple must `yield` results, not `return` them`` |
+| 5 | The columns you yield must match the output columns (step 3) | `expected but missing` or `unexpected field` |
+
+## How your code receives data
+
+Pick one row of this table. The class name in brackets and the method name must go together.
+
+| Class `(…)` | Method you write | You receive | Good for |
+|---|---|---|---|
+| `UDFOperatorV2` | `process_tuple(self, tuple_, port)` | one row at a time (use it like a dict: `tuple_["col"]`) | per-row formulas, filtering |
+| `UDFBatchOperator` | `process_batch(self, batch, port)` | `BATCH_SIZE` rows at a time, as a pandas DataFrame | calling an API in chunks |
+| `UDFTableOperator` | `process_table(self, table, port)` | **all** rows of one input, as a pandas DataFrame | sorting, statistics, ML training |
+| `UDFSourceOperator` (1-out UDF only) | `produce(self)` | nothing; you create the data | generating or downloading data |
+
+What you can `yield`:
+
+- **A row**: a `Tuple` or a `dict`, e.g. `yield {"name": "a", "value": 1}`.
+- **A table**: a pandas `DataFrame` or `Table`. Yield the DataFrame itself, not a dict that contains DataFrames.
+- **`None`**: produce nothing for this input.
+
+To drop a row, just don't `yield` it. To output several rows, `yield` several times.
+
+## One-time setup (loading a model, opening a file)
+
+Put setup code in `open()`, which runs once before any data arrives:
+
+```python
+from pytexera import *
+
+
+class ProcessTupleOperator(UDFOperatorV2):
+
+ @overrides
+ def open(self):
+ self.threshold = 100
+
+ @overrides
+ def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
+ if tuple_["Units Sold"] > self.threshold:
+ yield tuple_
+```
+
+If you write `__init__` instead, make its first line `super().__init__()`. Older Texera versions fail without it (see the table below).
+
+To set values like `threshold` from the property panel instead of in code, see [UI parameters](/docs/reference/operators/user-defined-functions/python/#ui-parameters).
+
+## Output columns
+
+The UDF's output columns are **Retain input columns** (the input's columns, if checked) plus **Extra output column(s)**. Every row you `yield` must contain exactly these columns.
+
+- Adding a column? Add it under **Extra output column(s)**.
+- Returning a completely new table (e.g. a summary)? Uncheck **Retain input columns** and list every output column.
+- With several inputs, **Retain input columns** keeps the columns of input 0 (for the 2-in UDF: the `tuples` input, port 1).
+
+## Several inputs
+
+Add input ports with the **+** under the operator's left side. Ports are numbered from 0. Your method's `port` argument tells you which input a row came from.
+
+- **Inputs arrive in no fixed order.** Don't assume input 0 finishes before input 1.
+- To combine inputs safely, use `UDFTableOperator`. It calls `process_table` once per input, when that input is complete. Keep each table and act when you have all of them:
+
+```python
+from pytexera import *
+
+NUM_INPUTS = 2
+
+
+class ProcessTableOperator(UDFTableOperator):
+
+ @overrides
+ def open(self):
+ self.tables = {}
+
+ @overrides
+ def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
+ self.tables[port] = table
+ if len(self.tables) == NUM_INPUTS:
+ yield self.tables[0].merge(self.tables[1], on="Country")
+```
+
+- To force an order, click an input port's dot. Under **dependencies**, list the ports that must finish first. For example, port 1 with dependencies `0` waits for port 0.
+- The **2-in Python UDF** has this built in: port 0 (`model`) is read completely before port 1 (`tuples`) starts.
+
+```python
+from pytexera import *
+
+
+class ProcessTupleOperator(UDFOperatorV2):
+
+ @overrides
+ def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
+ if port == 0: # model input: remember the model
+ self.model = tuple_["model"]
+ else: # tuples input: use it
+ tuple_["pred"] = self.model.predict([tuple_["text"]])[0]
+ yield tuple_
+```
+
+## Creating data (1-out Python UDF)
+
+```python
+from pytexera import *
+
+
+class GenerateOperator(UDFSourceOperator):
+
+ @overrides
+ def produce(self) -> Iterator[Union[TupleLike, TableLike, None]]:
+ for i in range(10):
+ yield {"number": i}
+```
+
+List every column (here `number`, type `integer`) under **Columns** in the property panel. More examples: [pytexera examples](https://github.com/apache/texera/tree/main/amber/src/main/python/pytexera/udf/examples).
+
+## Finding and fixing errors
+
+- `print(...)` in your code shows up in the **Console** tab. Click the UDF to see it.
+- If a row fails, the UDF turns **magenta** (paused) and the error appears in its **Console**. Fix the code, then stop the run (red ⊗ next to *Pause*) and run again.
+
+
Review Comment:
Thank you for reviewing this, some images were outdated and wrongly pushed. I will check and update it in the revised PR
##########
docs/tutorials/guide-for-how-to-use-texera/index.md:
##########
@@ -0,0 +1,117 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+---
+title: "Guide for how to use Texera"
+weight: 10
+---
+
+Texera lets you analyze data by connecting boxes (**operators**) on a canvas instead of writing a program. Each box does one step, like "read a file", "keep rows where…", or "compute an average". The arrows carry data from one box to the next.
+
+## Sign in
+
+Open the Texera address you were given in your browser and sign in.
+
+On a Texera you installed yourself ([Docker guide](/docs/getting-started/installing-using-docker/)), the built-in admin account is username `texera`, password `texera`.
+
+
+
+## The dashboard
+
+After signing in you see the dashboard. The left menu has:
+
+- **Hub**: workflows and datasets other people have made public.
+- **Your Work**
+ - **Workflows**: your workflows. Click **Create Workflow** to start one.
+ - **Datasets**: your data files ([how to upload](../create-dataset-upload-data/)).
+ - **Compute**: your computing units (see [below](#run-a-workflow)).
+ - **Environments**: extra Python packages for Python operators.
+ - **Quota**: how much storage and compute you have used.
+- **Admin**: users, executions and site settings. Admins only.
+
+## The workflow editor
+
+
+
+1. **Operators**: all available boxes, grouped by type. If the panel is closed, click the grid icon at the top left of the canvas. Drag a box onto the canvas, or type its name in the search box. Boxes added through search land on top of each other; drag them apart.
+2. **Canvas**: connect the output dot on the right of one box to the input dot on the left of the next.
+3. **Property panel**: click a box to set its options. A red outline means an option is missing or wrong.
+4. **Result panel**: shows a box's output, printed messages and errors. If it's closed, open it with the square icon at the bottom left.
+
+## Run a workflow
+
+A **computing unit** is the machine that runs your workflow. Pick one in the top bar, then click **Run**. If the list is empty, click **+ Computing Unit** to create one.
+
+
+
+- One computing unit can run all your workflows. Reuse it.
+- A new unit usually won't fix an error in the workflow itself (code or settings). It can help when the run needs more memory, CPU or GPU, or different Python packages.
+- Stopping (terminating) a unit deletes the results it holds. Your workflows and datasets are safe.
+
+More in [Computing Units](../computing-units/).
+
+## Read a running workflow
+
+Each box's outline color shows its state. The number on each side is how many rows went in (left) and came out (right).
+
+
+
+| Color | Meaning |
+|---|---|
+| Gray | Not started yet |
+| Yellow-green | Ready to start |
+| Orange | Running |
+| Magenta | Paused. You paused it, or a row caused an error |
+| Green | Finished |
+
+Some operators (e.g. Sort, Aggregate, and the build side of Join) must read **all** their input before they output anything. They show 0 rows out until the operator before them finishes. That's normal. See [How Texera runs your workflow](/docs/concepts/how-texera-runs-your-workflow/).
+
+## See the results
+
+1. Before running, click the box and turn on its **eye icon** (top toolbar). Texera shows results only for boxes with the eye on.
+2. After the run, click the box and open the result panel (square icon, bottom left) if it's closed. The **Result** tab shows its rows.
+
+The result panel only shows the box you clicked. If it says *No results available to display*, click a box that has the eye on.
+
+## Where to find errors
+
+- **Click an empty spot on the canvas.** If any operator failed, the result panel shows a **Static Error** tab with errors from every operator, including code that failed to start. A run that stops right away with every box at 0 rows usually means this.
+
+
Review Comment:
Thank you for reviewing this, some images were outdated and wrongly pushed. I will check and update it in the revised PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8737:
URL: https://github.com/apache/texera/pull/8737#issuecomment-5880512868
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/8737?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 92.73%. Comparing base ([`bcd0ec4`](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`84c238f`](https://app.codecov.io/gh/apache/texera/commit/84c238fa0c178bf08545ccb493d04ca9b9e0a2bc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #8737 +/- ##
============================================
- Coverage 92.73% 92.73% -0.01%
Complexity 4947 4947
============================================
Files 1241 1241
Lines 52608 52595 -13
Branches 6507 6503 -4
============================================
- Hits 48785 48772 -13
Misses 2206 2206
Partials 1617 1617
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.38% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `99.16% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.48% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.37% <ø> (-0.04%)` | :arrow_down: | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.37% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.53% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.55% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.73% <ø> (ø)` | | |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `98.50% <ø> (-0.01%)` | :arrow_down: | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/8737/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.09% <ø> (ø)` | | Carriedforward from [bcd0ec4](https://app.codecov.io/gh/apache/texera/commit/bcd0ec4f742264503114aa4388d82df8d35e9647?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/8737?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on code in PR #8737:
URL: https://github.com/apache/texera/pull/8737#discussion_r4138557702
##########
docs/tutorials/guide-to-use-python-udf/index.md:
##########
@@ -0,0 +1,210 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+---
+title: "Guide to Use a Python UDF"
+weight: 30
+---
+
+A **Python UDF** (user-defined function) is an operator that runs a small piece of Python code you write. Use it when no built-in operator does what you need.
+
+You don't need to be a programmer. Copy the example below and change one line at a time.
+
+## Which Python operator should I use?
+
+| You want to… | Use |
+|---|---|
+| Add or change a column with a one-line formula, e.g. `tuple_["Units Sold"] * 2` | **Python Lambda Function** (no class to write) |
+| Turn a whole table into one summary row, e.g. `table["Units Sold"].mean()` | **Python Table Reducer** (no class to write) |
+| Anything else with one or more inputs | **Python UDF** |
+| Apply a model (input 0) to data rows (input 1) | **2-in Python UDF** |
+| Create data from nothing (no input) | **1-out Python UDF** |
+
+## Your first UDF in 4 steps
+
+This example adds a `Revenue` column (units × price) to the sales data from [the dataset tutorial](../create-dataset-upload-data/).
+
+**1. Add the operator.** Drag **Python UDF** (under *User-defined Functions → Python*) onto the canvas and connect your data to it.
+
+**2. Write the code.** Click **Edit code content**. The editor opens with a template. Its classes are commented out (their lines start with `#`), so running it unchanged fails. Uncomment one class, or replace everything with:
+
+```python
+from pytexera import *
+
+
+class ProcessTupleOperator(UDFOperatorV2):
+
+ @overrides
+ def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
+ tuple_["Revenue"] = tuple_["Units Sold"] * tuple_["Unit Price"]
+ yield tuple_
+```
+
+
+
+**3. Declare the new column.** In the property panel, under **Extra output column(s)**, click **+** and add `Revenue` with type `double`. Leave **Retain input columns** checked to keep the original columns.
+
+<img src="udf-output-columns.png" alt="Retain input columns and one extra output column named Revenue" width="260">
+
+**4. Run.** Click the eye icon on the UDF, then **Run**. Click the UDF, then the **Result** tab in the result panel.
+
+## The 5 rules
+
+Almost every UDF problem comes from breaking one of these.
+
+| # | Rule | If you break it |
+|---|---|---|
+| 1 | The first line is `from pytexera import *` | `name 'UDFOperatorV2' is not defined` |
+| 2 | Exactly **one** class. Delete or comment out the others. | `There should be one and only one Operator defined` |
+| 3 | The method name must match the class you inherit from (table below) | `... No super class method found`, or rule 2's error |
+| 4 | Send results with `yield`, never `return` | ``process_tuple must `yield` results, not `return` them`` |
+| 5 | The columns you yield must match the output columns (step 3) | `expected but missing` or `unexpected field` |
+
+## How your code receives data
+
+Pick one row of this table. The class name in brackets and the method name must go together.
+
+| Class `(…)` | Method you write | You receive | Good for |
+|---|---|---|---|
+| `UDFOperatorV2` | `process_tuple(self, tuple_, port)` | one row at a time (use it like a dict: `tuple_["col"]`) | per-row formulas, filtering |
+| `UDFBatchOperator` | `process_batch(self, batch, port)` | `BATCH_SIZE` rows at a time, as a pandas DataFrame | calling an API in chunks |
+| `UDFTableOperator` | `process_table(self, table, port)` | **all** rows of one input, as a pandas DataFrame | sorting, statistics, ML training |
+| `UDFSourceOperator` (1-out UDF only) | `produce(self)` | nothing; you create the data | generating or downloading data |
+
+What you can `yield`:
+
+- **A row**: a `Tuple` or a `dict`, e.g. `yield {"name": "a", "value": 1}`.
+- **A table**: a pandas `DataFrame` or `Table`. Yield the DataFrame itself, not a dict that contains DataFrames.
+- **`None`**: produce nothing for this input.
+
+To drop a row, just don't `yield` it. To output several rows, `yield` several times.
+
+## One-time setup (loading a model, opening a file)
+
+Put setup code in `open()`, which runs once before any data arrives:
+
+```python
+from pytexera import *
+
+
+class ProcessTupleOperator(UDFOperatorV2):
+
+ @overrides
+ def open(self):
+ self.threshold = 100
+
+ @overrides
+ def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
+ if tuple_["Units Sold"] > self.threshold:
+ yield tuple_
+```
+
+If you write `__init__` instead, make its first line `super().__init__()`. Older Texera versions fail without it (see the table below).
+
+To set values like `threshold` from the property panel instead of in code, see [UI parameters](/docs/reference/operators/user-defined-functions/python/#ui-parameters).
+
+## Output columns
+
+The UDF's output columns are **Retain input columns** (the input's columns, if checked) plus **Extra output column(s)**. Every row you `yield` must contain exactly these columns.
+
+- Adding a column? Add it under **Extra output column(s)**.
+- Returning a completely new table (e.g. a summary)? Uncheck **Retain input columns** and list every output column.
+- With several inputs, **Retain input columns** keeps the columns of input 0 (for the 2-in UDF: the `tuples` input, port 1).
+
+## Several inputs
+
+Add input ports with the **+** under the operator's left side. Ports are numbered from 0. Your method's `port` argument tells you which input a row came from.
+
+- **Inputs arrive in no fixed order.** Don't assume input 0 finishes before input 1.
+- To combine inputs safely, use `UDFTableOperator`. It calls `process_table` once per input, when that input is complete. Keep each table and act when you have all of them:
+
+```python
+from pytexera import *
+
+NUM_INPUTS = 2
+
+
+class ProcessTableOperator(UDFTableOperator):
+
+ @overrides
+ def open(self):
+ self.tables = {}
+
+ @overrides
+ def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
+ self.tables[port] = table
+ if len(self.tables) == NUM_INPUTS:
+ yield self.tables[0].merge(self.tables[1], on="Country")
+```
+
+- To force an order, click an input port's dot. Under **dependencies**, list the ports that must finish first. For example, port 1 with dependencies `0` waits for port 0.
+- The **2-in Python UDF** has this built in: port 0 (`model`) is read completely before port 1 (`tuples`) starts.
+
+```python
+from pytexera import *
+
+
+class ProcessTupleOperator(UDFOperatorV2):
+
+ @overrides
+ def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]:
+ if port == 0: # model input: remember the model
+ self.model = tuple_["model"]
+ else: # tuples input: use it
+ tuple_["pred"] = self.model.predict([tuple_["text"]])[0]
+ yield tuple_
+```
+
+## Creating data (1-out Python UDF)
+
+```python
+from pytexera import *
+
+
+class GenerateOperator(UDFSourceOperator):
+
+ @overrides
+ def produce(self) -> Iterator[Union[TupleLike, TableLike, None]]:
+ for i in range(10):
+ yield {"number": i}
+```
+
+List every column (here `number`, type `integer`) under **Columns** in the property panel. More examples: [pytexera examples](https://github.com/apache/texera/tree/main/amber/src/main/python/pytexera/udf/examples).
+
+## Finding and fixing errors
+
+- `print(...)` in your code shows up in the **Console** tab. Click the UDF to see it.
+- If a row fails, the UDF turns **magenta** (paused) and the error appears in its **Console**. Fix the code, then stop the run (red ⊗ next to *Pause*) and run again.
+
+
Review Comment:
This screenshot shows `pampy.MatchError: '_' not provided`, which the table below lists as the older error for `return` instead of `yield`. With this PR a reader who makes that mistake sees the new message instead, so could you retake it with the PR's changes?
##########
docs/tutorials/guide-for-how-to-use-texera/index.md:
##########
@@ -0,0 +1,117 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+---
+title: "Guide for how to use Texera"
+weight: 10
+---
+
+Texera lets you analyze data by connecting boxes (**operators**) on a canvas instead of writing a program. Each box does one step, like "read a file", "keep rows where…", or "compute an average". The arrows carry data from one box to the next.
+
+## Sign in
+
+Open the Texera address you were given in your browser and sign in.
+
+On a Texera you installed yourself ([Docker guide](/docs/getting-started/installing-using-docker/)), the built-in admin account is username `texera`, password `texera`.
+
+
+
+## The dashboard
+
+After signing in you see the dashboard. The left menu has:
+
+- **Hub**: workflows and datasets other people have made public.
+- **Your Work**
+ - **Workflows**: your workflows. Click **Create Workflow** to start one.
+ - **Datasets**: your data files ([how to upload](../create-dataset-upload-data/)).
+ - **Compute**: your computing units (see [below](#run-a-workflow)).
+ - **Environments**: extra Python packages for Python operators.
+ - **Quota**: how much storage and compute you have used.
+- **Admin**: users, executions and site settings. Admins only.
+
+## The workflow editor
+
+
+
+1. **Operators**: all available boxes, grouped by type. If the panel is closed, click the grid icon at the top left of the canvas. Drag a box onto the canvas, or type its name in the search box. Boxes added through search land on top of each other; drag them apart.
+2. **Canvas**: connect the output dot on the right of one box to the input dot on the left of the next.
+3. **Property panel**: click a box to set its options. A red outline means an option is missing or wrong.
+4. **Result panel**: shows a box's output, printed messages and errors. If it's closed, open it with the square icon at the bottom left.
+
+## Run a workflow
+
+A **computing unit** is the machine that runs your workflow. Pick one in the top bar, then click **Run**. If the list is empty, click **+ Computing Unit** to create one.
+
+
+
+- One computing unit can run all your workflows. Reuse it.
+- A new unit usually won't fix an error in the workflow itself (code or settings). It can help when the run needs more memory, CPU or GPU, or different Python packages.
+- Stopping (terminating) a unit deletes the results it holds. Your workflows and datasets are safe.
+
+More in [Computing Units](../computing-units/).
+
+## Read a running workflow
+
+Each box's outline color shows its state. The number on each side is how many rows went in (left) and came out (right).
+
+
+
+| Color | Meaning |
+|---|---|
+| Gray | Not started yet |
+| Yellow-green | Ready to start |
+| Orange | Running |
+| Magenta | Paused. You paused it, or a row caused an error |
+| Green | Finished |
+
+Some operators (e.g. Sort, Aggregate, and the build side of Join) must read **all** their input before they output anything. They show 0 rows out until the operator before them finishes. That's normal. See [How Texera runs your workflow](/docs/concepts/how-texera-runs-your-workflow/).
+
+## See the results
+
+1. Before running, click the box and turn on its **eye icon** (top toolbar). Texera shows results only for boxes with the eye on.
+2. After the run, click the box and open the result panel (square icon, bottom left) if it's closed. The **Result** tab shows its rows.
+
+The result panel only shows the box you clicked. If it says *No results available to display*, click a box that has the eye on.
+
+## Where to find errors
+
+- **Click an empty spot on the canvas.** If any operator failed, the result panel shows a **Static Error** tab with errors from every operator, including code that failed to start. A run that stops right away with every box at 0 rows usually means this.
+
+
Review Comment:
This shows the old message, "There should be one and only one Operator defined", with nothing after it. With this PR the message also says what to fix, so please retake it with the PR's changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8737:
URL: https://github.com/apache/texera/pull/8737#issuecomment-5880558300
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline
> Compared against main `acc6dd1` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/36497181012)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| ⚪ | bs=10 sw=10 sl=64 | 397 | 0.243 | 24,429/33,071/33,071 us | ⚪ within ±5% / 🔴 +105.6% |
| 🔴 | bs=100 sw=10 sl=64 | 807 | 0.493 | 120,316/179,816/179,816 us | 🔴 +37.0% / 🔴 +66.7% |
| 🔴 | bs=1000 sw=10 sl=64 | 939 | 0.573 | 1,051,606/1,191,937/1,191,937 us | 🔴 +8.8% / 🔴 +15.1% |
<details><summary>Baseline details</summary>
Latest main `acc6dd1` from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 397 tuples/sec | 405 tuples/sec | 807.63 tuples/sec | -2.0% | -50.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.243 MB/s | 0.247 MB/s | 0.493 MB/s | -1.6% | -50.7% |
| bs=10 sw=10 sl=64 | p50 | 24,429 us | 25,117 us | 12,578 us | -2.7% | +94.2% |
| bs=10 sw=10 sl=64 | p95 | 33,071 us | 34,478 us | 16,088 us | -4.1% | +105.6% |
| bs=10 sw=10 sl=64 | p99 | 33,071 us | 34,478 us | 19,009 us | -4.1% | +74.0% |
| bs=100 sw=10 sl=64 | throughput | 807 tuples/sec | 873 tuples/sec | 1,039 tuples/sec | -7.6% | -22.3% |
| bs=100 sw=10 sl=64 | MB/s | 0.493 MB/s | 0.533 MB/s | 0.634 MB/s | -7.5% | -22.3% |
| bs=100 sw=10 sl=64 | p50 | 120,316 us | 114,367 us | 101,098 us | +5.2% | +19.0% |
| bs=100 sw=10 sl=64 | p95 | 179,816 us | 131,219 us | 107,890 us | +37.0% | +66.7% |
| bs=100 sw=10 sl=64 | p99 | 179,816 us | 131,219 us | 119,853 us | +37.0% | +50.0% |
| bs=1000 sw=10 sl=64 | throughput | 939 tuples/sec | 950 tuples/sec | 1,065 tuples/sec | -1.2% | -11.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.573 MB/s | 0.58 MB/s | 0.65 MB/s | -1.2% | -11.9% |
| bs=1000 sw=10 sl=64 | p50 | 1,051,606 us | 1,052,389 us | 992,940 us | -0.1% | +5.9% |
| bs=1000 sw=10 sl=64 | p95 | 1,191,937 us | 1,095,657 us | 1,035,756 us | +8.8% | +15.1% |
| bs=1000 sw=10 sl=64 | p99 | 1,191,937 us | 1,095,657 us | 1,062,329 us | +8.8% | +12.2% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,503.38,200,128000,397,0.243,24429.26,33071.29,33071.29
1,100,10,64,20,2478.13,2000,1280000,807,0.493,120316.25,179816.43,179816.43
2,1000,10,64,20,21293.54,20000,12800000,939,0.573,1051606.37,1191937.19,1191937.19
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: explain Python UDF mistakes and rewrite user docs for non-programmers [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8737:
URL: https://github.com/apache/texera/pull/8737#issuecomment-5880468586
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Committers with relevant context:** `@parshimers`
You can request their reviews formally with `/request-review @parshimers`.
- **Contributors with relevant context:** `@aicam`, `@carloea2`, `@aglinxinyuan`
You can notify them by mentioning `@aicam`, `@carloea2`, `@aglinxinyuan` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org