You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "carloea2 (via GitHub)" <gi...@apache.org> on 2026/09/09 20:40:56 UTC
[I] Support Arrow batches as an explicit Python UDF input and output mode [texera]
carloea2 opened a new issue, #8476:
URL: https://github.com/apache/texera/issues/8476
### Feature Summary
Allow Python UDFs to explicitly receive and return Arrow batches, avoiding the intermediate Python Tuple conversion while keeping existing UDF APIs unchanged.
PyAmber currently expands incoming Arrow tables into Tuples and converts UDF table output back through Tuples before rebuilding Arrow. This adds substantial overhead for operators that already work on entire batches.
An exploratory local conversion benchmark on upstream commit ec3a9dd3ca, using PyArrow 23.0.1 and 10,000 rows with 10 string columns of 64-character values, measured:
| Conversion path | Time |
| --- | --- |
| Current Arrow, Tuple, pandas, Tuple, Arrow path | 1,566 ms |
| Direct Arrow, pandas, Arrow path | 20.4 ms |
This is approximately 77 times faster for the measured conversion path. It indicates potential savings from avoiding row conversion, not a measured 77 times improvement in workflow execution. The proposed engine path has not been implemented. The full Arrow Flight benchmark was blocked locally by a JOOQ schema mismatch, so an end-to-end benchmark is still needed.
### Proposed Solution or Design
Introduce an explicit opt-in API, for example:
```python
import pyarrow as pa
import pyarrow.compute as pc
from pyamber import ArrowBatchOperator # Proposed API
class FilterPrices(ArrowBatchOperator):
BATCH_SIZE = 4096
def process_batch(self, batch: pa.Table, port: int):
yield batch.filter(pc.greater(batch["price"], 100))
```
The engine would deliver Arrow batches directly and accept Arrow output without expanding every row into a Tuple. Syntax alone would not remove the current input and output conversions.
- Keep existing TupleOperatorV2, BatchOperator, and TableOperator behavior unchanged.
- Respect configured batch sizes, final partial batches, port boundaries, and row order.
- Preserve schema validation, Texera partition hashing, storage writes, and backpressure.
- Define inspection and retry around a batch invocation for the new API, with control handling between invocations and output yields. Do not automatically rerun UDFs to fall back, since they may have side effects.
- Specify null, timestamp, binary, and error behavior explicitly, including what happens to valid output preceding a validation failure.
Start with ArrowBatchOperator. A separate ArrowTableOperator could later support whole-port input at completion. Validate the proposal with reproducible conversion and full engine benchmarks, plus tests for control handling and output equivalence where the APIs share semantics.
### Affected Area
Workflow Engine (Amber)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support Arrow batches as an explicit Python UDF input and output mode [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on issue #8476:
URL: https://github.com/apache/texera/issues/8476#issuecomment-5608450649
@Yicong-Huang @chenlica
Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support Arrow batches as an explicit Python UDF input and output mode [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 closed issue #8476: Support Arrow batches as an explicit Python UDF input and output mode
URL: https://github.com/apache/texera/issues/8476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support Arrow batches as an explicit Python UDF input and output mode [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8476:
URL: https://github.com/apache/texera/issues/8476#issuecomment-5609566057
<!-- texera:template-compliance -->
👋 Thanks for opening this issue, @carloea2!
It looks like the issue description doesn't quite follow our template yet:
- This issue doesn't appear to use one of our templates (Bug, Feature, or Task). Please open it using a template so the required details are captured.
Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed.
You can find the template prompts by editing the description, or see [CONTRIBUTING.md](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) for the full contribution flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support Arrow batches as an explicit Python UDF input and output mode [texera]
Posted by "carloea2 (via GitHub)" <gi...@apache.org>.
carloea2 commented on issue #8476:
URL: https://github.com/apache/texera/issues/8476#issuecomment-5609175949
Moved to the Ideas discussion: https://github.com/apache/texera/discussions/8477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org