[PR] fix(core): convert an Avro map to an Arrow map [hudi-rs]

["linliu-code (via GitHub)" <gi...@apache.org>]

linliu-code opened a new pull request, #664:
URL: https://github.com/apache/hudi-rs/pull/664

   **Stacked on #639–#663** — review only the last commit.
   
   ## The bug
   
   `avro_to_arrow/schema.rs` modelled an Avro map as:
   
   ```rust
   DataType::Dictionary(Box::new(DataType::Utf8), Box::new(value_type))
   ```
   
   **An Arrow dictionary key must be an integer type.** `Dictionary(Utf8, V)` is not a valid Arrow type — not merely an odd choice. And it cannot reconcile against the `Map` a parquet base file actually carries:
   
   ```
   evolution: unsupported container combination
     Map("key_value": non-null Struct("key": non-null Utf8, "value": Int32), unsorted)
     -> Dictionary(Utf8, Int32) for field 'map_null_val'
   ```
   
   So any table with a map column fails as soon as a log block has to merge with its base file. **This affects the existing reader too** — it is a shared conversion, and a map column has never been readable through either.
   
   ## The fix
   
   An Avro map becomes `Map(key_value: struct<key: string, value: V>, sorted=false)` — the same shape and the same entry-field name the parquet reader produces, so the two agree by name.
   
   On the array side, `build_map_array` constructs a real `MapArray`. Entries are materialized as two-field records (`key`, `value`) so the existing struct machinery builds both children; that is also why `child_schema_lookup` now registers those two positions, since a struct-valued map needs its own fields resolvable underneath them.
   
   Entries are emitted **in key order**. Avro maps are unordered and the Arrow type says `sorted = false`, but a stable order keeps a read reproducible rather than dependent on hash iteration.
   
   ## What it unblocks
   
   `harness_null_container_elements` — maps and lists containing NULL elements — now passes. That was the hardest of the three map cases.
   
   The other two (`harness_all_data_types`, `harness_mixed_column_types`) still fail, but **on something else**: a decimal column in a log block reads as NULL where the Spark snapshot has a value. The map error was failing those cases earlier and hiding it. They stay pinned, now carrying that finding instead.
   
   This also removes the reason base-file-only slices are routed away from the merge-on-read engine in #659 — that guard exists purely because every parquet fixture here has a map column.
   
   ## Tests
   
   Two direct conversion tests: a map of primitives produces `Map` with a non-nullable `key_value` entry struct and a non-nullable `Utf8` key; a map of records keeps the value as a nested struct rather than collapsing it.
   
   Plus the harness case above, which exercises the array builder end to end against a Spark snapshot.
   
   Full workspace green: 1178 lib + 79 table-read + 39 datafusion + 21 + 12. Ignored 8 → 7.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(core): convert an Avro map to an Arrow map [hudi-rs]

["linliu-code (via GitHub)" <gi...@apache.org>]

linliu-code closed pull request #664: fix(core): convert an Avro map to an Arrow map
URL: https://github.com/apache/hudi-rs/pull/664


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(core): convert an Avro map to an Arrow map [hudi-rs]

["linliu-code (via GitHub)" <gi...@apache.org>]

linliu-code commented on PR #664:
URL: https://github.com/apache/hudi-rs/pull/664#issuecomment-5302609907

   Superseded by #660, which merged as `75b188e`. This PR's content is in that commit — the ten PRs in this series were the reviewable decomposition of the port, and shipping went through the squashed #660 instead.
   
   Every review thread here was answered before that merge; the replies stay on the thread for the record. Closing to keep the queue honest about what is still in flight.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org