[I] Column stats on MOR table version 9 crashes on map/array-nested leaf columns during log-append [hudi]

["vinishjail97 (via GitHub)" <gi...@apache.org>]

vinishjail97 opened a new issue, #19125:
URL: https://github.com/apache/hudi/issues/19125

   **Describe the problem you faced**
   
   Building the column-stats index on a table version 9 (timeline layout V2) **Merge-on-Read** table crashes during the inline log-append upsert when the col-stats column list contains a leaf nested inside a `MAP` or `ARRAY` (e.g. `my_map.key_value.value`, `my_array.list.element`):
   
   ```
   org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
     ...
   Caused by: java.lang.IllegalStateException: Cannot get field from schema type: MAP
   	at org.apache.hudi.common.schema.HoodieSchema.getField(HoodieSchema.java:1172)
   	at org.apache.hudi.avro.AvroRecordContext.getFieldValueFromIndexedRecord(AvroRecordContext.java:90)
   	at org.apache.hudi.common.model.HoodieAvroIndexedRecord.getColumnValueAsJava(HoodieAvroIndexedRecord.java:193)
   	at org.apache.hudi.metadata.HoodieTableMetadataUtil.collectColumnRangeFieldValueV2(HoodieTableMetadataUtil.java:354)
   	at org.apache.hudi.metadata.HoodieTableMetadataUtil.collectColumnRangeMetadata(HoodieTableMetadataUtil.java:296)
   	at org.apache.hudi.io.HoodieInlineLogAppendHandle.collectColumnStats(HoodieInlineLogAppendHandle.java:182)
   	at org.apache.hudi.io.HoodieAppendHandle.processAppendResult(HoodieAppendHandle.java:454)
   ```
   
   **Root cause**
   
   #17694 added support for column stats on primitives nested inside `MAP`/`ARRAY` by teaching the **schema-side** navigator (`HoodieSchema.getNestedField`) and the base-file (Parquet) path to resolve the Parquet-style synthetic accessors `.key_value.key`, `.key_value.value` and `.list.element`. Such leaves therefore pass `isColumnTypeSupported` (the resolved leaf is a scalar).
   
   The **column-stats V2 record path** used by MOR inline log-append (`HoodieTableMetadataUtil.collectColumnRangeFieldValueV2` -> `HoodieRecord.getColumnValueAsJava` -> `AvroRecordContext.getFieldValueFromIndexedRecord`) was never taught the same navigation. It splits the field path on `.` and assumes every segment is a `RECORD` field, so when it reaches the `MAP`/`ARRAY` schema it calls `HoodieSchema.getField(...)` on it, which throws `IllegalStateException: Cannot get field from schema type: MAP`.
   
   The V1 path (table version 6) does not hit this: `HoodieAvroUtils.getNestedFieldVal` returns `null` when a path segment is not a record, so version 6 tolerates these column-stats columns (they simply contribute nothing from the record path, and get their statistics from the Parquet base-file footer). Only the V2 record navigator was left asymmetric.
   
   **To Reproduce**
   
   1. Create a MOR table at table version 9 with column stats enabled and a col-stats column list that includes a map/array leaf, e.g. `my_map.key_value.value` or `my_array.list.element`.
   2. Insert some records, then upsert records into the same file group (forces a MOR log-append update).
   3. The upsert fails with `HoodieUpsertException` caused by `IllegalStateException: Cannot get field from schema type: MAP`.
   
   This surfaces in Apache XTable when a MOR source table at table version 9 has column stats configured on map/array leaves.
   
   **Expected behavior**
   
   Collecting column stats on a MOR table at version 9 should succeed for map/array-nested leaf columns, matching version 6 behavior. The record-path value navigator should degrade to `null` for a path it cannot resolve to a single value (statistics for such leaves are still collected from the base-file/Parquet footer path, which is the coverage #17694 added), instead of throwing.
   
   **Environment Description**
   
   * Hudi version : master (1.3.0-SNAPSHOT)
   * Spark version : 3.5
   * Table type : Merge-on-Read
   * Table version : 9 (col-stats index V2)
   
   **Additional context**
   
   Fix mirrors `HoodieAvroUtils.getNestedFieldVal`: `AvroRecordContext.getFieldValueFromIndexedRecord` returns `null` when a path segment cannot be resolved as a plain `RECORD` field (a `MAP`/`ARRAY` intermediate or a null intermediate) rather than calling `getField` on a non-record schema. A map/array leaf is multi-valued per record and has no single value to fold into a min/max, so `null` (no stats from the record path) is the correct and safe result.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Column stats on MOR table version 9 crashes on map/array-nested leaf columns during log-append [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous closed issue #19125: Column stats on MOR table version 9 crashes on map/array-nested leaf columns during log-append
URL: https://github.com/apache/hudi/issues/19125


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org