You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/09/23 00:24:23 UTC
[D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
GitHub user aglinxinyuan created a discussion: Use a loop's variables in any operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, you can type `$K` into a property of any operator in the body, and every iteration runs that operator with the current value of the loop variable `K`.
Issues: #8635, #8636, #8641. PRs: #8639 (engine), #8640 (property panel), #8642 (Python operators).
---
## Why
A loop declares its variables in Loop Start's **Initialization** (for example `K = 2`) and updates them in Loop End's **Update** (`K += 1`). Today only those two operators can read the variables. The operators in between, the ones you actually loop for, take constants. So a parameter sweep such as "run the same pipeline for K = 2, 3, ..., 10" has to be rewired by hand or rewritten as a Python UDF.
## What it looks like
Example: which reviews pass each rating threshold.
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
1. Declare the variable in Loop Start's Initialization: `min_stars = 1`.
2. Open any operator between Loop Start and Loop End. In a property, type `$`: the panel suggests the variables of the enclosing loop, here `$min_stars`. Pick one.
3. Run. The first iteration filters with `stars >= 1`, the second with `stars >= 2`, and so on up to 5. Loop End collects the rows of every iteration.
The operator itself does not change. That is the point: it works for built-in operators that were written long before loops existed.
### What you see in the property panel
| Situation | What happens |
|---|---|
| Operator inside a loop | Text, number and true/false fields accept `$name`. Typing `$` suggests the loop's variables. |
| `$foo`, but no enclosing loop declares `foo` | The field is flagged: "$foo is not a variable of an enclosing block". The workflow cannot run until it is fixed. |
| The variable is renamed in Loop Start | Operators still using the old name are flagged right away. |
| Operator moved out of the loop | Its fields go back to normal. `$K` in a number field is an error again, and a text such as `$AAPL` is plain text, exactly as today. |
### In your own Python UDF
For a UDF, the variable belongs in your code rather than in a property. A UDF inside the loop receives the loop's variables when they arrive, the same way it receives each tuple:
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_state(self, state, port):
self.min_stars = state["min_stars"] # called when the loop's variables arrive
return state
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.min_stars:
yield tuple_
```
With #8642, the same values are also available anywhere in the UDF as `self.loop_state`.
## Not supported yet
- **Operators that generate Python code**, such as the sklearn and chart operators, and the property fields of Python UDFs. The panel accepts `$K` there, but the run stops with "cannot refer to loop variables yet".
- **An operator that also takes input from outside the loop**, for example a join whose other side comes from before Loop Start.
- **Properties that name a column**, such as the attributes of Projection.
- **Text that starts with `$` inside a loop.** A cashtag such as `$AAPL` always means a variable there. Outside loops nothing changes.
- **Operators inside an inner loop** of nested loops. The panel suggests the variables of both loops there, but the run does not bind them reliably yet.
- **The Form View** does not offer `$K` yet.
## Feedback wanted
1. Is `$name` the right syntax? And how should a literal `$` be written inside a loop, for example `$$AAPL`?
2. Inside a loop, true/false fields become text boxes so that they can hold `$flag`. Would you rather keep the checkbox and add a "use a loop variable" toggle?
3. Should the panel refuse `$K` up front on operators that cannot use it yet, instead of the run reporting it?
4. Which operators should come next? The sklearn operators are the obvious candidates, for example K-means with `$K` clusters.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a discussion: Use a loop's variables in any operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, type `$i` into a property of any operator, and every iteration runs that operator with the loop's current `i`.
**Pull requests** (reviewed and merged separately):
| Part | Pull request | Issue |
|---|---|---|
| Engine: binds `$i` in operator properties | [#8639 feat(amber): bind $i loop-variable references in operators](https://github.com/apache/texera/pull/8639) | #8635 |
| Property panel: accepts and suggests `$i` | [#8640 feat(frontend): accept $variable references in loop blocks](https://github.com/apache/texera/pull/8640) | #8636 |
| Python operators: `self.state` in UDFs | [#8642 feat(pyamber): register the state before process_state](https://github.com/apache/texera/pull/8642) | #8641 |
## How it works
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
Declare the variable in Loop Start, type `$min_stars` into a property of any operator in the loop, and run. Each iteration uses its own value, and the operator's code does not change.
## Screenshots
| Outside a loop (as today) | Inside a loop | Undeclared name |
|---|---|---|
|  |  |  |
A run of `Text Input (1..6) -> Loop Start (i = 0) -> Limit ($i) -> Loop End`: iteration `i` lets `i` rows through, so Loop End collects 0 + 1 + ... + 5 = 15 rows.
## In a Python UDF
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_state(self, state, port):
self.min_stars = state["min_stars"] # the loop's variables arrive here
return state
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.min_stars:
yield tuple_
```
## Not supported yet
Python-generated operators (sklearn, charts), operators that also take input from outside the loop, properties that name a column, a literal `$...` text inside a loop, operators in an inner loop of nested loops, and the Form View.
## Feedback wanted
1. Is `$name` the right syntax, and how should a literal `$` be written inside a loop?
2. Inside a loop, true/false fields become text boxes. Would a checkbox with a "use a loop variable" toggle be better?
3. Which operators should come next? The sklearn K-means, with `$i` as its number of clusters, is the obvious one.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $i in a property [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica added a comment to the discussion: Use a loop's variables in any operator inside the loop: type $i in a property
For 3, a good use case is the "k" in the k-means algorithm. We want the user of the algorithm to access the k value, which can be passed as a state variable from the LoopStart operator.
GitHub link: https://github.com/apache/texera/discussions/8643#discussioncomment-18618131
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a discussion: Use a loop's variables in any operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, type `$i` into a property of any operator, and every iteration runs that operator with the loop's current `i`.
**Pull requests** (reviewed and merged separately):
| Part | Pull request | Issue |
|---|---|---|
| Engine: binds `$i` in operator properties | [#8639 feat(amber): bind $i loop-variable references in operators](https://github.com/apache/texera/pull/8639) | #8635 |
| Property panel: accepts and suggests `$i` | [#8640 feat(frontend): accept $variable references in loop blocks](https://github.com/apache/texera/pull/8640) | #8636 |
| Python operators: `self.loop_state` in UDFs | [#8642 feat(pyamber): register loop_state before process_state](https://github.com/apache/texera/pull/8642) | #8641 |
## How it works
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
Declare the variable in Loop Start, type `$min_stars` into a property of any operator in the loop, and run. Each iteration uses its own value, and the operator's code does not change.
## Screenshots
| Outside a loop (as today) | Inside a loop | Undeclared name |
|---|---|---|
|  |  |  |
A run of `Text Input (1..6) -> Loop Start (i = 0) -> Limit ($i) -> Loop End`: iteration `i` lets `i` rows through, so Loop End collects 0 + 1 + ... + 5 = 15 rows.
## In a Python UDF
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_state(self, state, port):
self.min_stars = state["min_stars"] # the loop's variables arrive here
return state
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.min_stars:
yield tuple_
```
## Not supported yet
Python-generated operators (sklearn, charts), operators that also take input from outside the loop, properties that name a column, a literal `$...` text inside a loop, operators in an inner loop of nested loops, and the Form View.
## Feedback wanted
1. Is `$name` the right syntax, and how should a literal `$` be written inside a loop?
2. Inside a loop, true/false fields become text boxes. Would a checkbox with a "use a loop variable" toggle be better?
3. Which operators should come next? The sklearn K-means, with `$i` as its number of clusters, is the obvious one.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a discussion: Use a loop's variables in any operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, type `$K` into a property of any operator, and every iteration runs that operator with the loop's current `K`.
**Pull requests** (reviewed and merged separately):
| Part | Pull request | Issue |
|---|---|---|
| Engine: binds `$K` in operator properties | [#8639 feat(amber): bind $K loop-variable references in operators](https://github.com/apache/texera/pull/8639) | #8635 |
| Property panel: accepts and suggests `$K` | [#8640 feat(frontend): accept $variable references in loop blocks](https://github.com/apache/texera/pull/8640) | #8636 |
| Python operators: `self.loop_state` in UDFs | [#8642 feat(pyamber): register loop_state before process_state](https://github.com/apache/texera/pull/8642) | #8641 |
## How it works
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
Declare the variable in Loop Start, type `$min_stars` into a property of any operator in the loop, and run. Each iteration uses its own value, and the operator's code does not change.
## Screenshots
| Outside a loop (as today) | Inside a loop | Undeclared name |
|---|---|---|
|  |  |  |
A run of `Text Input (1..6) -> Loop Start (i = 0) -> Limit ($i) -> Loop End`: iteration `i` lets `i` rows through, so Loop End collects 0 + 1 + ... + 5 = 15 rows.
## In a Python UDF
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_state(self, state, port):
self.min_stars = state["min_stars"] # the loop's variables arrive here
return state
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.min_stars:
yield tuple_
```
## Not supported yet
Python-generated operators (sklearn, charts), operators that also take input from outside the loop, properties that name a column, a literal `$...` text inside a loop, operators in an inner loop of nested loops, and the Form View.
## Feedback wanted
1. Is `$name` the right syntax, and how should a literal `$` be written inside a loop?
2. Inside a loop, true/false fields become text boxes. Would a checkbox with a "use a loop variable" toggle be better?
3. Which operators should come next? The sklearn K-means with `$K` clusters is the obvious one.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a discussion: Use a loop's variables in any operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, you can type `$K` into a property of any operator in the body, and every iteration runs that operator with the current value of the loop variable `K`.
**Pull requests** (reviewed and merged separately; the panel needs the engine first):
| Part | Pull request | Issue |
|---|---|---|
| Engine: binds `$K` in operator properties | [#8639 feat(amber): bind $K loop-variable references in operators](https://github.com/apache/texera/pull/8639) | #8635 |
| Property panel: accepts and suggests `$K` | [#8640 feat(frontend): accept $variable references in loop blocks](https://github.com/apache/texera/pull/8640) | #8636 |
| Python operators: `self.loop_state` in UDFs | [#8642 feat(pyamber): register loop_state before process_state](https://github.com/apache/texera/pull/8642) | #8641 |
---
## Why
A loop declares its variables in Loop Start's **Initialization** (for example `K = 2`) and updates them in Loop End's **Update** (`K += 1`). Today only those two operators can read the variables. The operators in between, the ones you actually loop for, take constants. So a parameter sweep such as "run the same pipeline for K = 2, 3, ..., 10" has to be rewired by hand or rewritten as a Python UDF.
## What it looks like
Example: which reviews pass each rating threshold.
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
1. Declare the variable in Loop Start's Initialization: `min_stars = 1`.
2. Open any operator between Loop Start and Loop End. In a property, type `$`: the panel suggests the variables of the enclosing loop, here `$min_stars`. Pick one.
3. Run. The first iteration filters with `stars >= 1`, the second with `stars >= 2`, and so on up to 5. Loop End collects the rows of every iteration.
The operator itself does not change. That is the point: it works for built-in operators that were written long before loops existed.
### What you see in the property panel
| Situation | What happens |
|---|---|
| Operator inside a loop | Text, number and true/false fields accept `$name`. Typing `$` suggests the loop's variables. |
| `$foo`, but no enclosing loop declares `foo` | The field is flagged: "$foo is not a variable of an enclosing block". The workflow cannot run until it is fixed. |
| The variable is renamed in Loop Start | Operators still using the old name are flagged right away. |
| Operator moved out of the loop | Its fields go back to normal. `$K` in a number field is an error again, and a text such as `$AAPL` is plain text, exactly as today. |
### In your own Python UDF
For a UDF, the variable belongs in your code rather than in a property. A UDF inside the loop receives the loop's variables when they arrive, the same way it receives each tuple:
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_state(self, state, port):
self.min_stars = state["min_stars"] # called when the loop's variables arrive
return state
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.min_stars:
yield tuple_
```
With #8642, the same values are also available anywhere in the UDF as `self.loop_state`.
## Not supported yet
- **Operators that generate Python code**, such as the sklearn and chart operators, and the property fields of Python UDFs. The panel accepts `$K` there, but the run stops with "cannot refer to loop variables yet".
- **An operator that also takes input from outside the loop**, for example a join whose other side comes from before Loop Start.
- **Properties that name a column**, such as the attributes of Projection.
- **Text that starts with `$` inside a loop.** A cashtag such as `$AAPL` always means a variable there. Outside loops nothing changes.
- **Operators inside an inner loop** of nested loops. The panel suggests the variables of both loops there, but the run does not bind them reliably yet.
- **The Form View** does not offer `$K` yet.
## Feedback wanted
1. Is `$name` the right syntax? And how should a literal `$` be written inside a loop, for example `$$AAPL`?
2. Inside a loop, true/false fields become text boxes so that they can hold `$flag`. Would you rather keep the checkbox and add a "use a loop variable" toggle?
3. Should the panel refuse `$K` up front on operators that cannot use it yet, instead of the run reporting it?
4. Which operators should come next? The sklearn operators are the obvious candidates, for example K-means with `$K` clusters.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a discussion: Use a loop's variables in any operator inside the loop: type $K in a property
> **In one sentence:** inside a Loop Start ... Loop End block, type `$i` into a property of any operator, and every iteration runs that operator with the loop's current `i`.
**Pull requests** (reviewed and merged separately):
| Part | Pull request | Issue |
|---|---|---|
| Engine: binds `$i` in operator properties | [#8639 feat(amber): bind $i loop-variable references in operators](https://github.com/apache/texera/pull/8639) | #8635 |
| Property panel: accepts and suggests `$i` | [#8640 feat(frontend): accept $variable references in loop blocks](https://github.com/apache/texera/pull/8640) | #8636 |
| Python operators: `self.state` in UDFs | [#8642 feat(pyamber): register the state before process_state](https://github.com/apache/texera/pull/8642) | #8641 |
| Python-generated operators (sklearn, charts): bind `$i` in their properties | [#8671 feat: bind $i in generated Python operators' properties](https://github.com/apache/texera/pull/8671) | #8669 |
## How it works
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
Declare the variable in Loop Start, type `$min_stars` into a property of any operator in the loop, and run. Each iteration uses its own value, and the operator's code does not change.
## Design
A reference such as `$i` is a placeholder that is filled in when the loop runs:
```
compile workflow starts state message arrives first tuple
$i is recorded -> the operator holds -> the loop's variables reach -> the placeholder takes i's value,
as a placeholder a placeholder the operator used for every tuple of the iteration
```
| Step | What happens | PR |
|---|---|---|
| Parse | A numeric or boolean property holding `$i` gets a typed placeholder (0, 0.0 or false), and the reference is recorded beside the operator. A text property keeps `$i` as its text. | #8639 |
| Compile | The compiler finds the operators inside each loop. Inside, the reference is kept; outside every loop, a text such as `$AAPL` stays plain text, and a numeric `$i` is an error. | #8639 |
| Run, Java operators | The loop's variables reach every operator in the loop as a state message, before the iteration's data. The operator is built at its first tuple, with each placeholder replaced by the variable's value. | #8639 |
| Run, Python operators | The worker registers the state message on the operator as `self.state` before it calls `process_state`, so every callback can read the variables. | #8642 |
| Python-generated operators | Their code is written from the properties before the loop runs. A text reference becomes a read of the variable when the code runs. For a number or boolean, the code is generated twice from two sample values; where the two versions differ by exactly the value, the code reads the variable instead. This covers every generated operator without changing any of them. | #8671 |
| Property panel | Finds the loops that enclose an operator, turns its fields into text boxes that suggest the loops' variables, and flags a name that no enclosing loop declares. | #8640 |
Only Loop Start's initialization and Loop End's update change the variables, so they stay fixed within an iteration. Every copy of the state that an operator receives therefore agrees, and filling each placeholder once per iteration is enough.
## Screenshots
| Outside a loop (as today) | Inside a loop | Undeclared name |
|---|---|---|
|  |  |  |
A run of `Text Input (1..6) -> Loop Start (i = 0) -> Limit ($i) -> Loop End`: iteration `i` lets `i` rows through, so Loop End collects 0 + 1 + ... + 5 = 15 rows.
## In a Python UDF
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.state["min_stars"]: # the loop's current variables
yield tuple_
```
## Not supported yet
Operators that also take input from outside the loop, properties that name a column, a numeric or boolean property that a Python-generated operator checks or computes with before it runs, a literal `$...` text inside a loop, operators in an inner loop of nested loops, and the Form View.
## Feedback wanted
1. Is `$name` the right syntax, and how should a literal `$` be written inside a loop?
2. How should a user refer to a loop variable in place of a true/false checkbox, such as the one below? A checkbox cannot hold `$name`, so inside a loop it currently becomes a text box that takes `true`, `false` or `$name`. Would a checkbox with a "use a loop variable" toggle be better?
<img width="200" alt="A true/false checkbox in the property panel" src="https://github.com/user-attachments/assets/296059c1-5314-4be4-a0df-e971ad5ad7e5" />
3. What are good use cases for loop variables? For example: sweeping a model's hyperparameter, a filter threshold, or a chart setting across iterations.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $i in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a discussion: Use a loop's variables in any operator inside the loop: type $i in a property
> **In one sentence:** inside a Loop Start ... Loop End block, type `$i` into a property of any operator, and every iteration runs that operator with the loop's current `i`.
**Pull requests** (reviewed and merged separately):
| Part | Pull request | Issue |
|---|---|---|
| Engine: binds `$i` in operator properties | [#8639 feat(amber): bind $i loop-variable references in operators](https://github.com/apache/texera/pull/8639) | #8635 |
| Property panel: accepts and suggests `$i` | [#8640 feat(frontend): accept $variable references in loop blocks](https://github.com/apache/texera/pull/8640) | #8636 |
| Python operators: `self.state` in UDFs | [#8642 feat(pyamber): register the state before process_state](https://github.com/apache/texera/pull/8642) | #8641 |
| Python-generated operators (sklearn, charts): bind `$i` in their properties | [#8671 feat: bind $i in generated Python operators' properties](https://github.com/apache/texera/pull/8671) | #8669 |
## How it works
```mermaid
flowchart LR
R[Reviews] --> LS["Loop Start<br/>Initialization: min_stars = 1<br/>Output: table"]
LS --> F["Filter<br/>stars >= $min_stars"]
F --> LE["Loop End<br/>Update: min_stars += 1<br/>Condition: min_stars <= 5"]
```
Declare the variable in Loop Start, type `$min_stars` into a property of any operator in the loop, and run. Each iteration uses its own value, and the operator's code does not change.
## Design
A reference such as `$i` is a placeholder that is filled in when the loop runs:
```
compile workflow starts state message arrives first tuple
$i is recorded -> the operator holds -> the loop's variables reach -> the placeholder takes i's value,
as a placeholder a placeholder the operator used for every tuple of the iteration
```
| Step | What happens | PR |
|---|---|---|
| Parse | A numeric or boolean property holding `$i` gets a typed placeholder (0, 0.0 or false), and the reference is recorded beside the operator. A text property keeps `$i` as its text. | #8639 |
| Compile | The compiler finds the operators inside each loop. Inside, the reference is kept; outside every loop, a text such as `$AAPL` stays plain text, and a numeric `$i` is an error. | #8639 |
| Run, Java operators | The loop's variables reach every operator in the loop as a state message, before the iteration's data. The operator is built at its first tuple, with each placeholder replaced by the variable's value. | #8639 |
| Run, Python operators | The worker registers the state message on the operator as `self.state` before it calls `process_state`, so every callback can read the variables. | #8642 |
| Python-generated operators | Their code is written from the properties before the loop runs. A text reference becomes a read of the variable when the code runs. For a number or boolean, the code is generated twice from two sample values; where the two versions differ by exactly the value, the code reads the variable instead. This covers every generated operator without changing any of them. | #8671 |
| Property panel | Finds the loops that enclose an operator, turns its fields into text boxes that suggest the loops' variables, and flags a name that no enclosing loop declares. | #8640 |
Only Loop Start's initialization and Loop End's update change the variables, so they stay fixed within an iteration. Every copy of the state that an operator receives therefore agrees, and filling each placeholder once per iteration is enough.
## Screenshots
| Outside a loop (as today) | Inside a loop | Undeclared name |
|---|---|---|
|  |  |  |
A run of `Text Input (1..6) -> Loop Start (i = 0) -> Limit ($i) -> Loop End`: iteration `i` lets `i` rows through, so Loop End collects 0 + 1 + ... + 5 = 15 rows.
## In a Python UDF
```python
class ProcessTupleOperator(UDFOperatorV2):
def process_tuple(self, tuple_, port):
if tuple_["stars"] >= self.state["min_stars"]: # the loop's current variables
yield tuple_
```
## Not supported yet
Operators that also take input from outside the loop, properties that name a column, a numeric or boolean property that a Python-generated operator checks or computes with before it runs, a literal `$...` text inside a loop, operators in an inner loop of nested loops, and the Form View.
## Feedback wanted
1. Is `$name` the right syntax, and how should a literal `$` be written inside a loop?
2. How should a user refer to a loop variable in place of a true/false checkbox, such as the one below? A checkbox cannot hold `$name`, so inside a loop it currently becomes a text box that takes `true`, `false` or `$name`. Would a checkbox with a "use a loop variable" toggle be better?
<img width="200" alt="A true/false checkbox in the property panel" src="https://github.com/user-attachments/assets/296059c1-5314-4be4-a0df-e971ad5ad7e5" />
3. What are good use cases for loop variables? For example: sweeping a model's hyperparameter, a filter threshold, or a chart setting across iterations.
GitHub link: https://github.com/apache/texera/discussions/8643
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan edited a comment on the discussion: Use a loop's variables in any operator inside the loop: type $K in a property
For 2, we basically want to understand how users should reference a state variable in place of a boolean checkbox, like the one shown here:
<img width="200" alt="image" src="https://github.com/user-attachments/assets/296059c1-5314-4be4-a0df-e971ad5ad7e5" />
For 3, I’m trying to understand what some good use cases for this state variable would be.
I intentionally focused on the user experience in the current version. I can add a separate section with more detail on the underlying design and implementation.
GitHub link: https://github.com/apache/texera/discussions/8643#discussioncomment-18609001
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
GitHub user Xiao-zhen-Liu added a comment to the discussion: Use a loop's variables in any operator inside the loop: type $K in a property
1. I think `$name` is OK, as long as we use a reserved symbol that can be easily parsed to indicate this is a loop variable. Which specific syntax is an implementation detail.
2. Can you clarify this question means? I don't understand it. Why would true/false fields become text boxes inside a loop?
3. Can all the Python native operators share the same code? If so it does not matter which operator comes next.
In general, can you try to include a bit more detail about this topic, especially what the design is? Currently only the user experience is described.
GitHub link: https://github.com/apache/texera/discussions/8643#discussioncomment-18608890
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org
Re: [D] Use a loop's variables in any operator inside the loop: type $K in a property [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
GitHub user aglinxinyuan added a comment to the discussion: Use a loop's variables in any operator inside the loop: type $K in a property
For 2, we basically want to understand how users should reference a state variable in place of a boolean checkbox, like the one shown here:
<img width="556" height="866" alt="image" src="https://github.com/user-attachments/assets/296059c1-5314-4be4-a0df-e971ad5ad7e5" />
For 3, I’m trying to understand what some good use cases for this state variable would be.
I intentionally focused on the user experience in the current version. I can add a separate section with more detail on the underlying design and implementation.
GitHub link: https://github.com/apache/texera/discussions/8643#discussioncomment-18609001
----
This is an automatically sent email for dev@texera.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@texera.apache.org