Re: [I] Spark 4 variant shredding: FIXED typed_value broken on both write and read paths [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous commented on issue #19442:
URL: https://github.com/apache/hudi/issues/19442#issuecomment-6165989504

   **Leaf type map for `Spark4VariantShreddingProvider`** (master `61dde86`, from code reading; not run)
   
   - **Write:** `HoodieAvroWriteSupport` shreds a variant into the `typed_value` leaf declared by the Avro schema (`avroTypeToScalarType`, then `convertScalarToAvro`).
   - **Read:** `HoodieVariantReconstruction` rebuilds the variant from a base file. The leaf's Avro type comes from the footer converter, then the same `avroTypeToScalarType` and the `AvroVariantRow` getters feed Spark's `ShreddingUtils.rebuild`.
   
   ### Write: Avro `typed_value` type, then ScalarType, then the value handed to the Avro writer
   
   ```mermaid
   flowchart LR
       WOK["boolean, int, long, float, double, string, bytes<br/>decimal on bytes or fixed, date<br/>timestamp-micros, local-timestamp-micros<br/>uuid on string"] --> WOKS["matching ScalarType"] --> WOKR["OK"]
       WF["fixed, no logical type"] --> WFS["BinaryType"] --> WFV["byte[] wrapped as ByteBuffer"] --> WFR["ClassCastException:<br/>ByteBuffer is not GenericFixed<br/>size never checked"]
       WU["uuid on fixed(16)"] --> WUS["UuidType"] --> WUV["String from UUID.toString()"] --> WUR["UnresolvedUnionException"]
       WT["time-millis on int<br/>time-micros on long"] --> WTS["IntegralType<br/>no time branch"] --> WTV["Integer or Long"] --> WTR["writes, but variant integers<br/>land in a TIME column"]
       WD["timestamp-millis, local-timestamp-millis<br/>enum, map"] --> WDS["null: not shredded"] --> WDR["value stays in the residual<br/>by design"]
   
       classDef ok fill:#bbf7d0,stroke:#15803d,color:#111827
       classDef bad fill:#fecaca,stroke:#b91c1c,color:#111827
       classDef wrong fill:#fde68a,stroke:#b45309,color:#111827
       classDef skip fill:#e5e7eb,stroke:#4b5563,color:#111827
       class WOK,WOKS,WOKR ok
       class WF,WFS,WFV,WFR,WU,WUS,WUV,WUR bad
       class WT,WTS,WTV,WTR wrong
       class WD,WDS,WDR skip
   ```
   
   ### Read: parquet `typed_value` type, then Avro type from the footer converter, then ScalarType, then the rebuilt value
   
   ```mermaid
   flowchart LR
       ROK["boolean, int32, int64, float, double<br/>string, binary, DATE<br/>DECIMAL on BINARY or FLBA, FLBA needs PR 20324<br/>TIMESTAMP(MICROS), UUID on FLBA(16)"] --> ROKS["matching ScalarType"] --> ROKR["OK"]
       RF["FLBA, no annotation"] --> RFA["fixed"] --> RFS["BinaryType"] --> RFR["getBinary casts to ByteBuffer:<br/>ClassCastException on GenericFixed"]
       RD["DECIMAL on INT32 or INT64"] --> RDA["int or long<br/>converter drops DECIMAL"] --> RDS["IntegralType"] --> RDR["silent wrong value:<br/>unscaled integer, 1.50 becomes 150"]
       RT["TIME(MILLIS) or TIME(MICROS)"] --> RTA["time-millis or time-micros"] --> RTS["IntegralType"] --> RTR["time rebuilt as an integer"]
       RN["TIMESTAMP(NANOS)"] --> RNA["long<br/>converter drops it"] --> RNS["IntegralType"] --> RNR["timestamp rebuilt as an integer"]
       RM["TIMESTAMP(MILLIS)"] --> RMA["timestamp-millis"] --> RMS["null: no ScalarType"] --> RMR["NPE in Spark rebuild<br/>when typed_value is non-null"]
   
       classDef ok fill:#bbf7d0,stroke:#15803d,color:#111827
       classDef bad fill:#fecaca,stroke:#b91c1c,color:#111827
       classDef wrong fill:#fde68a,stroke:#b45309,color:#111827
       class ROK,ROKS,ROKR ok
       class RF,RFA,RFS,RFR,RD,RDA,RDS,RDR,RM,RMA,RMS,RMR bad
       class RT,RTA,RTS,RTR,RN,RNA,RNS,RNR wrong
   ```
   
   Green: correct. Red: fails or returns a wrong value. Amber: no error, but the leaf's type is lost. Grey: not shredded, by design.
   
   | Leaf | Direction | Outcome | Reachable from |
   |---|---|---|---|
   | `fixed`, no logical type | write | ClassCastException, no size check | Hand-declared Avro schema only |
   | uuid on `fixed(16)` | write | UnresolvedUnionException | Hand-declared Avro schema only |
   | FLBA, no annotation | read | ClassCastException | Files written by other engines |
   | DECIMAL on INT32 / INT64 | read | Silent wrong value (unscaled integer) | Files written by other engines; Hudi writes decimals as FLBA |
   | TIME (millis / micros) | write and read | Stored and rebuilt as integers | Hand-declared Avro schema; other engines' files |
   | TIMESTAMP(NANOS) | read | Rebuilt as an integer | Files written by other engines |
   | TIMESTAMP(MILLIS) | read | NPE in `ShreddingUtils.rebuild` | Files written by other engines |
   
   Notes:
   - The first three rows are this issue's report; the rest turned up while mapping. Only the DECIMAL on INT32 / INT64 row returns a wrong value without an error.
   - The read-side uuid failure in the original report (`GenericFixed` in `getUuid`) is not reachable through files: the footer converter maps FLBA(16) UUID to an Avro string, and parquet-avro's `FieldUUIDConverter` reads it as one.
   - Declining plain `fixed` and uuid on `fixed` at write time covers the first two rows only. For DECIMAL on INT and NANOS, the converter has already dropped the annotation, so the provider cannot see it. TIME keeps its Avro logical type, but Spark's `VariantSchema` has no time scalar to map it to.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org