You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "Yohahaha (via GitHub)" <gi...@apache.org> on 2026/09/29 06:33:38 UTC
[I] [spark] Support columnar batch reads for Arrow log tables [fluss]
Yohahaha opened a new issue, #4509:
URL: https://github.com/apache/fluss/issues/4509
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Split the "batch read instead of row read" task out of the Spark scan optimization umbrella issue #3213.
Spark currently consumes Arrow-format Fluss log tables through row readers. Support columnar batch reads for bounded, pure-Fluss log-table scans using Spark `ColumnarBatch`.
The implementation should preserve offset and time boundaries, filter progress, projection (including zero-column counts), limits, Spark type semantics, and resource cleanup. Keep existing fallback readers for non-Arrow formats and lake union scans. Streaming and primary-key table batch reads are outside this task's scope.
Validate with Spark 3.5 unit and integration tests. Implementation PR: #4507.
### Willingness to contribute
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org