You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "gyang94 (via GitHub)" <gi...@apache.org> on 2026/09/10 12:33:37 UTC

[PR] [kafka] Transcode mapped raw and string records to Arrow [fluss]

gyang94 opened a new pull request, #4285:
URL: https://github.com/apache/fluss/pull/4285

   <!-- Generated-by: Codex following https://github.com/apache/fluss/blob/main/AGENTS.md -->
   
   ### Purpose
   
   Closes #4282. Part of #4185.
   
   Use the DDL mapping contract to decode raw/string keys and values, assemble physical rows with optional timestamps and ordered headers, and encode owned native Arrow log bytes.
   
   This is PR06, depending on #4284 (PR05).
   
   ### Brief change log
   
   - Raw bytes and strict UTF-8 strings map to the configured physical columns.
   - Nullable values, empty bytes, NOT NULL violations and duplicate/nullable headers are handled correctly.
   - The native batch uses the table schema ID and remains readable after Arrow resources close.
   - Arrow writers and allocators close on encoding failures.
   
   Incremental scope: 6 files, +640 / -0 (314 implementation and 326 test lines). The implementation contract and its failure-path tests are kept together as one review unit; the line count includes Javadoc and license headers.
   
   ### Tests
   
   Java 11 at `503136295d6bbaed81b7e7845dceb2d64db2d9d4`:
   
   - `mvn -o -pl fluss-kafka spotless:apply clean verify`: 75 unit tests and 2 integration tests passed.
   - Checkstyle, Spotless, RAT and `git diff --check` passed.
   - The two existing integration tests cover Kafka transport and native DDL mapping; this stage does not activate production Produce.
   
   The full repository test suite and Flink SQL runtime were not run.
   
   ### API and Format
   
   Adds raw/string-to-Arrow conversion for the existing DDL contract, with no wire or storage format change. JSON decoding and append integration remain separate.
   
   ### Stack and review
   
   Draft while prerequisites are unmerged. The PR targets Apache `main`, so the full Files changed view includes prerequisites. Use [this incremental comparison](https://github.com/gyang94/fluss/compare/936fd3f39e9e6fe595ed6bf8e30f0e2b5185ffe3...503136295d6bbaed81b7e7845dceb2d64db2d9d4) to review this stage alone.
   
   Generative AI disclosure: Codex assisted with extraction, implementation, tests and commit organization. Human review is required before merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [kafka] Transcode binary and string Produce records to Arrow [fluss]

Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong merged PR #4285:
URL: https://github.com/apache/fluss/pull/4285


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org