You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:42:39 UTC

[I] Revisit ENUM handling in Spark parquet reader and writer [hudi]

hudi-bot opened a new issue, #17324:
URL: https://github.com/apache/hudi/issues/17324

   Spark Avro reader logic treats Avro ENUM type as String when converted to Spark StructType schema, which is converted back to Avro String type if it goes through Row to GenericRecord conversion.  This is different from our previous logic where the record is directly read out as Avro GenericRecord.
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8660
   - Type: Sub-task
   - Parent: https://issues.apache.org/jira/browse/HUDI-9108
   - Fix version(s):
     - 1.1.0
   
   
   ---
   
   
   ## Comments
   
   12/May/25 19:44;linliu;Where does it translate ROW to GenericRecord? 
   
   Can we use the original AVRO schema? If not, why?;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Revisit ENUM handling in Spark parquet reader and writer [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17324:
URL: https://github.com/apache/hudi/issues/17324#issuecomment-5365823864

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8660).
   
   **Findings: still open.**
   
   The asymmetry described - Avro `ENUM` treated as `String` when converted to a Spark `StructType`, then converted back to Avro `String` rather than `ENUM` on the Row-to-GenericRecord path, unlike the older direct `GenericRecord` read - has no reworking commit that I could find.
   
   A corroborating signal that ENUM is still second-class: in `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java`, both `isColumnTypeSupportedV1` and `isColumnTypeSupportedV2` explicitly exclude `HoodieSchemaType.ENUM`, so ENUM columns cannot be indexed for column stats on either index version. That is consistent with the type being lossily flattened on the Spark path.
   
   Since this is a schema-fidelity question rather than a crash, the useful first step is a decision on intended semantics: should a round trip preserve `ENUM`, or is flattening to `String` the accepted behaviour with the symbol constraint documented as lost? The patch shape follows from that. Given it is marked blocker and In Progress, the assignee may already have a view worth recording here.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org