You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Ma77Ball (via GitHub)" <gi...@apache.org> on 2026/09/17 03:09:03 UTC
[PR] feat(amber): columnar join, vectorized sink, Python Arrow passthrough [texera]
Ma77Ball opened a new pull request, #8560:
URL: https://github.com/apache/texera/pull/8560
### What changes were proposed in this PR?
- Third PR of the columnar series: adds the columnar join, offloads the sink decode, lets terminal operators consume columns, and passes Arrow straight through to Python.
- `HashJoinProbeOpExec` probes the hash table directly from the Arrow batch (selective probe), decoding only the matched rows.
- Sink offload: `OutputManager.saveArrowBatchToStorageIfNeeded` hands the Arrow batch to the result-writer thread (`OutputPortStorageWriterThread`), moving the decode off the DP thread.
- Terminal/sink operators return `Consumed` for a columnar batch instead of forcing a row decode.
- Python passthrough: a `ColumnarFrame` reaching `PythonProxyClient` is streamed straight to Arrow Flight with no tuple round-trip (Python already speaks Arrow).
- Contract: `ColumnarResult` gains `EmitRows` and `processColumnarBatch` takes a `port`, so an operator can consume columns but emit rows.
### Any related issues, documentation, discussions?
Part of: #8556
### How was this PR tested?
- `sbt "testOnly *ColumnarShuffleCorrectnessSpec"`: 2-worker shuffle, row output == columnar output through the join and aggregate.
- `sbt "testOnly *OutputPortStorageWriterThreadSpec"`: the Arrow batch write path stores the same rows as the row path.
- `sbt "testOnly *ProjectionOpExecSpec *VectorizedFilterCorrectnessSpec"`: operator output unchanged under the new `port` contract.
### Was this PR authored or co-authored using generative AI tooling?
Co-authored with Claude Opus 4.8 (Anthropic) in compliance with ASF policy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): columnar join, vectorized sink, Python Arrow passthrough [texera]
Posted by "github-actions[bot] (via GitHub) via notifications" <no...@texera.apache.org>.
github-actions[bot] commented on PR #8560:
URL: https://github.com/apache/texera/pull/8560#issuecomment-6566730152
This pull request has been automatically closed due to 37 days of inactivity.
If you would like to continue this work, please re-open the PR or open a new
one referencing this. Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): columnar join, vectorized sink, Python Arrow passthrough [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8560:
URL: https://github.com/apache/texera/pull/8560#issuecomment-5707872589
<!-- texera-reviewer-suggestion -->
### Automated Reviewer Suggestions
Based on the `git blame` history of the changed files, we recommend the following reviewers:
- **Contributors with relevant context:** `@aglinxinyuan`, `@Yicong-Huang`, `@eugenegujing`
You can notify them by mentioning `@aglinxinyuan`, `@Yicong-Huang`, `@eugenegujing` in a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): columnar join, vectorized sink, Python Arrow passthrough [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8560:
URL: https://github.com/apache/texera/pull/8560#issuecomment-6257763682
This pull request has been automatically marked as **stale** because it has
not had any activity for 30 days. It will be closed in **7 days** if no
further activity occurs.
If this PR is still relevant, please:
- Push new commits or rebase onto the latest `main`, or
- Comment with a status update, or
- Address any open review feedback.
To exempt this PR from auto-close, a maintainer can add the `pinned` label.
Thank you for your contribution to Apache Texera (incubating)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): columnar join, vectorized sink, Python Arrow passthrough [texera]
Posted by "github-actions[bot] (via GitHub) via notifications" <no...@texera.apache.org>.
github-actions[bot] closed pull request #8560: feat(amber): columnar join, vectorized sink, Python Arrow passthrough
URL: https://github.com/apache/texera/pull/8560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): columnar join, vectorized sink, Python Arrow passthrough [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8560:
URL: https://github.com/apache/texera/pull/8560#issuecomment-5707901687
<!-- texera-benchmarks-comment -->
## ๐ Arrow Flight E2E bench
_(no arrow-flight-e2e.csv in artifact)_
[Full dashboard](https://apache.github.io/texera/dev/bench/) ยท [Workflow run](https://github.com/apache/texera/actions/runs/35177080035)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(amber): columnar join, vectorized sink, Python Arrow passthrough [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #8560:
URL: https://github.com/apache/texera/pull/8560#issuecomment-5707887194
### :warning: JUnit XML file not found
The CLI was unable to find any JUnit XML files to upload.
For more help, visit our [troubleshooting guide](https://docs.codecov.com/docs/test-analytics#troubleshooting).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org