You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "hhvu0102 (via GitHub)" <gi...@apache.org> on 2025/11/21 21:07:37 UTC
[I] Function is called multiple times when using multi ports [texera]
hhvu0102 opened a new issue, #4074:
URL: https://github.com/apache/texera/issues/4074
### What happened?
When there are multiple ports, each port is called twice. For example, using this code:
```
class ProcessTableOperator(UDFTableOperator):
@overrides
def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
df0 = pd.DataFrame(table)
print(f'port is {port}')
df1 = pd.DataFrame(table)
print(f'port is {port}')
yield None
```
The `f'port is {port}` is printed twice
<img width="774" height="311" alt="Image" src="https://github.com/user-attachments/assets/2915907d-6a3b-4e12-a7fc-edd7d0fc1598" />
This signals to me that the function is called as many times as the number of ports, which should not be a normal behavior in my opinion.
Additionally, `port 1` is called before `port 0` because `port 1` data is smaller and is , which Chris and Jiadong mentioned is what's intended. However, if an user is unaware, they may write a code to read in the data from `port 0` first because that's the intended order, which will be an error as data is not loaded. Currently I write my own wrapper so inputs from each port can be recognized correctly, but do you have any example of how you often handle this case? I can make a new issue for this question if that's easier.
### How to reproduce?
Dataset: `samn16081314-downsampled-5k`; workflow: `PanKbase_example_5k_multi-port` - I have shared these via email to Meng, Jiadong, Chris and Chen. Please let me know if I should share them with any other team members!
To run this workflow, you'll need to install some packages yourself. In Python:
```
pip install pysam
pip install cellbender
pip install scikit-image
```
In R:
```
if (!require("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("DropletUtils")
```
### Version
1.1.0-incubating (Pre-release/Master)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Function is called multiple times when using multi ports [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on issue #4074:
URL: https://github.com/apache/texera/issues/4074#issuecomment-5900325986
@hhvu0102 Thanks for the follow-up questions.
**1. Using the data from both ports at once**
Xinyuan answered this in #8683: with the dependency from port 1 to port 0, call your function in the port 1 branch. If you'd rather not rely on the order, keep each table and call the function when both have arrived. `process_table` is called once for each port, and only after all of that port's data has arrived, so you don't need any code to check that the data has finished loading:
```python
from pytexera import *
class ProcessTableOperator(UDFTableOperator):
@overrides
def open(self) -> None:
self.tables = {}
@overrides
def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
self.tables[port] = table
if len(self.tables) == 2:
# both ports have all their data; this runs once
yield your_function(self.tables[0], self.tables[1])
```
Both ways assume "Worker count" is 1, which is the default. With more workers, each port's data is split among the workers, and each worker would call `your_function` with only its part of the data.
**2. The wiki example**
You're right: the example only works because port 0 finishes before port 1, and the wiki doesn't point that out. It is written for the "2-in Python UDF" operator, where port 1 (`tuples`) always waits until port 0 (`model`) has finished, so `self.model` is set before any tuple arrives on port 1. In a regular Python UDF with two ports there is no such wait unless you add the dependency, so the example fails when port 1's data comes first. It also fails if port 0 gets no tuples at all. The Python UDF guide is being rewritten in #8737, and the new version says that inputs arrive in no fixed order and shows the pattern above.
**3. Why it works this way**
Texera uses pipelined execution: operators run at the same time, and data is passed to the next operator while it is being produced, not after the previous operator finishes. This has three benefits: results start to appear sooner, operators work in parallel instead of waiting for each other, and intermediate results don't have to be stored in full between operators. To make this work, each operator, built-in or UDF, receives its input one tuple at a time, together with the port the tuple came in on. That is the Tuple API (`process_tuple`). The Table API is built on top of it: it collects each port's tuples and calls `process_table` when that port has no more data, so it is called once per port.
For the same reasons, the engine doesn't wait for all ports by default: an operator can start on one input while the operators feeding its other input are still running. When an operator does need one input to be complete before another, it declares a dependency between the two ports, and the engine finishes the first port before it sends any data to the second. The built-in Hash Join works this way: it builds a hash table from the whole left input, then looks up each right tuple in it. The 2-in Python UDF has this dependency built in, and in the regular Python UDF you can add it yourself, as in the GIF above. Without a dependency, the order depends on which upstream operator finishes first, which is why port 1 came first in your workflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Function is called multiple times when using multi ports [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on issue #4074:
URL: https://github.com/apache/texera/issues/4074#issuecomment-3572160296
@hhvu0102 Thank you for the issue. You are trying to use some advanced features of the UDF. What you described all look like intended behavior to me.
For the first "print" problem:
`process_table(self, table: Table, port: int)` is `invoked once for each port`, and the `port` parameter is a single value for each invocation. In your case, the `process_table` for port 1 is invoked first, and since you code is printing it twice in the `process_table` function, it will be printed twice. The behavior of your code is:
- port 1's logic, which is processed first:
`df0` = `port 1`'s table
`print port 1`
`df1` = `port 1`'s table
`print port 1`
- port 1's logic, which is processed first:
`df0` = `port 0`'s table
`print port 0`
`df1` = `port 0`'s table
`print port 0`
The correct way to write these kind of logic should be (similar to what we mentioned in the [wiki]([url](https://github.com/apache/texera/wiki/Guide-to-Use-a-Python-UDF#2-in-udf))):
```
@overrides
def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
if port == 0:
# port 0's processing logic
elif port == 1:
# port 1's processing logic
```
For the second problem about input order:
You can specify the order of the two inputs by specifying a dependency from port 1 to port 0 (see the below gif). This will tell the engine to process port 0 before port 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Function is called multiple times when using multi ports [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #4074:
URL: https://github.com/apache/texera/issues/4074#issuecomment-3583452257
Closing as it is not a bug. @hhvu0102 please feel free to reopen this if you have more questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Function is called multiple times when using multi ports [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed issue #4074: Function is called multiple times when using multi ports
URL: https://github.com/apache/texera/issues/4074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Function is called multiple times when using multi ports [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed issue #4074: Function is called multiple times when using multi ports
URL: https://github.com/apache/texera/issues/4074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Function is called multiple times when using multi ports [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on issue #4074:
URL: https://github.com/apache/texera/issues/4074#issuecomment-3565742118
@hhvu0102 Thank you.
@Xiao-zhen-Liu Can you take a look when you have a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Function is called multiple times when using multi ports [texera]
Posted by "hhvu0102 (via GitHub)" <gi...@apache.org>.
hhvu0102 commented on issue #4074:
URL: https://github.com/apache/texera/issues/4074#issuecomment-5839420911
Hi all,
I'd like to reopen this thread with some follow-up questions:
1) If I want to use the data from both ports at the same time, what is your recommendation to write my function? Pseudo-code that I imagine is:
```
# some code to make sure that data from both ports are present and have finished loading -- not sure what your recommendation is
def function_to_called_once_only_when_all_args_available(a, b):
print(f"function_to_call Called Once with: {a}, {b}")
return {"a": a, "b": b}
```
2) The example you supplied on [the wiki](https://github.com/apache/texera/wiki/Guide-to-Use-a-Python-UDF) may break if `self.model` in the `else` statement does not exist yet. To me, this example implicitly assumed there's a `model` in `self` which is not the case in all instances.
3) I'm curious what the motivation for this design choice is. Would be great if you can explain, even if it involves complicated technical details.
Thanks team!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org