[PR] feat: deserialize DataSplit from Paimon-native bytes (core + C binding) [paimon-rust]

["JunRuiLee (via GitHub)" <gi...@apache.org>]

JunRuiLee opened a new pull request, #565:
URL: https://github.com/apache/paimon-rust/pull/565

   ### Purpose
   
   `DataSplit` already supports serialization to the Java-native wire format — `DataSplit::serialize()` (raw v8, byte-compatible with Java `DataSplit#serialize`) and `DataSplit::serialize_split_v1()` (the `SplitSerializer` v1 frame), both golden-verified — but there is no inverse. paimon-rust cannot reconstruct a `DataSplit` from bytes.
   
   This blocks the planning-vs-reading separation where paimon-rust is the **reader**: a planner in another process/engine serializes a `DataSplit` and ships it to a worker, which must deserialize it before reading. This PR adds that direction, so paimon-rust can serve as a read kernel behind an external planner.
   
   ### Brief change log
   
   - **core**: add the inverse of the existing serialization, built on existing primitives (`BinaryRow::from_serialized_bytes` + typed getters) — no new low-level infra:
     - `DataFileMeta::from_serialized_row_data` — reverse of `to_serialized_row_data` (fixed 20-field `BinaryRow`).
     - `DataSplit::deserialize` — reverse of `serialize()` (raw v8 body); consumes the whole buffer (trailing bytes rejected).
     - `DataSplit::deserialize_split_v1` — reverse of `serialize_split_v1()` (`SplitSerializer` frame): type 1 `DataSplit` and type 3 `IndexedSplit` (with row ranges); other type ids return `Unsupported`.
     - Leaf decoders: binary arrays (string/bigint), `SimpleStats`, Java modified-UTF, deletion-file list, and big-endian cursor readers.
   - **C binding**: `paimon_plan_from_split_bytes(data, len)` — build a one-split plan from raw-v8 `DataSplit` bytes, wrapped in the existing `paimon_plan` (usable by `paimon_table_read_to_arrow`, freed by `paimon_plan_free`).
   - Malformed/truncated input yields typed errors (`DataInvalid` / `Unsupported`) and never panics; count/length prefixes are bounded against the remaining buffer to avoid unbounded allocation. v8 only for now (current Java `DataSplit.VERSION`), structured so a future v9 branch is a localized addition.
   
   ### Tests
   
   - `cargo test -p paimon` — round-trip against the existing `datasplit_v8` / `split_v1_data` / `split_v1_indexed` goldens, symmetric `deserialize(serialize(x)) == x`, Java modified-UTF edge cases (NUL, non-ASCII, surrogate pairs, malformed continuation bytes, truncation), unsupported type ids, inverted row ranges, trailing bytes, and huge-count anti-abort cases.
   - `cargo test -p paimon-c` — `paimon_plan_from_split_bytes` round-trip (serialize a planned split, deserialize via the C entry, read), plus null/empty and garbage error paths.
   
   ### API and Format
   
   - Adds core methods and one C API symbol; no change to existing symbols or to any storage format. Deserialization is byte-compatible with the existing serialization.
   
   ### Documentation
   
   - Doc comments on each new function; no separate docs needed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat: deserialize DataSplit from Paimon-native bytes (core + C binding) [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #565:
URL: https://github.com/apache/paimon-rust/pull/565#issuecomment-5030096234

      - `deserialize_binary_array_long` directly calls `push(None)` on null elements without first verifying the required 8-byte fixed slot for each element. A small input forged with an “extremely large count + all-null bitmap + no element slots” will be accepted and result in approximately 128-fold memory amplification; this entry point is exposed to C calls, which may cause the process to run out of memory (OOM). A validation check for `header + count * 8 <= data.len()` should be performed before the loop.
      - Additionally, `beforeDeletionFiles` is silently discarded when it is non-null, whereas Java would reject such a split. It is recommended to maintain compatibility and fail loudly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat: deserialize DataSplit from Paimon-native bytes (core + C binding) [paimon-rust]

["JunRuiLee (via GitHub)" <gi...@apache.org>]

JunRuiLee commented on PR #565:
URL: https://github.com/apache/paimon-rust/pull/565#issuecomment-5030666910

   Thanks for the review, both are good catches — fixed.
   
   1. **Binary-array amplification** — `deserialize_binary_array_str/long` now validate that the fixed element region (`count * 8`) fits within the buffer up front (`check_binary_array_fits`), before allocating or iterating. The `long` decoder was the reachable one: null elements skip the per-slot bounds check, so a forged "large count + all-null bitmap + no element slots" input could push `count` `None`s from a tiny buffer. The up-front check closes that; added a regression test with an all-null bitmap and no slots.
   
   2. **`beforeDeletionFiles`** — now rejected with `Unsupported` when non-null instead of being discarded, matching Java `DataSplit#deserialize` which throws in that case (and mirroring the existing `beforeFiles != 0` handling).
   
   While there, I also made `isStreaming = true` fail loudly for the same reason: it's a semantic bit Java readers branch on, and this decoder only serves batch splits (`isStreaming = false`), so silently dropping it would be wrong.
   
   Updated in `fc0f99c` and `748c7b3`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat: deserialize DataSplit from Paimon-native bytes (core + C binding) [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #565:
URL: https://github.com/apache/paimon-rust/pull/565


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org