You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "Yohahaha (via GitHub)" <gi...@apache.org> on 2026/09/29 06:28:47 UTC

[PR] [spark] Support columnar batch reads for Arrow log tables [fluss]

Yohahaha opened a new pull request, #4507:
URL: https://github.com/apache/fluss/pull/4507

   <!-- Generated-by: OpenAI Codex (GPT-6) following https://github.com/apache/fluss/blob/main/AGENTS.md -->
   
   ### Purpose
   
   Related to #3213: implements the "batch read instead of row read" item.
   
   Bounded Spark scans currently read Arrow-format Fluss log tables row by row. This change exposes their Arrow batches as Spark `ColumnarBatch` values, enabling columnar execution for pure-Fluss log-table batch scans.
   
   ### Brief change log
   
   - Select a dedicated columnar reader for bounded Arrow log scans, preserving existing readers for non-Arrow formats, primary-key tables, lake union scans, and streaming.
   - Adapt primitive and nested Arrow vectors to Spark types, including temporal precision conversion, reordered projections, and zero-column batches for `COUNT(*)`. Projection is applied in Spark because the client batch polling API does not support server-side projection.
   - Enforce offset and time boundaries and limits, account for filtered scan progress, and release Arrow batches and client resources on completion, early closure, and errors.
   
   ### Tests
   
   - Validated with local Maven 3.9.16 and JDK 11 against Spark 3.5 only; Spark 3.4 was not tested.
   - All 304 tests across 24 Spark suites passed before rebasing, including log reads, time ranges, primary-key reads, streaming regression coverage, writes, catalogs, and Paimon/Iceberg lake reads. Integration tests ran outside the sandbox to permit local sockets.
   - New unit coverage: `FlussAppendColumnarReaderTest` (9 tests) and `FlussArrowColumnVectorTest` (7 tests), covering boundaries, filtered progress, limits, cleanup, projection, nulls, nested types, and temporal conversion using real Arrow batches.
   - `SparkLogTableReadTest` checks columnar execution, reordered projection, zero-column counts, and the non-Arrow fallback.
   - Spark module build, Spotless/Checkstyle, and standalone RAT checks passed. RAT excluded only environment-owned `.agents/**` and `.codex/**` directories.
   
   ### API and Format
   
   No public Fluss API or storage format changes. No new dependencies or configuration options. Eligible bounded Arrow log scans select columnar reads automatically.
   
   ### Documentation
   
   No documentation changes; the existing Spark batch-read interface and SQL syntax are unchanged.
   
   Generative AI disclosure: OpenAI Codex (GPT-6) assisted implementation and tests; GPT-6-luna executed Maven validation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org