You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 10:59:59 UTC

[I] Revisit how we decide the reader schema for reading log files in the filegroup reader [hudi]

hudi-bot opened a new issue, #16694:
URL: https://github.com/apache/hudi/issues/16694

   This is motivated by the information loss when converting between spark and avro schemas and enums turn into strings. This requires a rewrite of the record in HUDI-8412. 
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8415
   - Type: Task
   - Epic: https://issues.apache.org/jira/browse/HUDI-6243
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Revisit how we decide the reader schema for reading log files in the filegroup reader [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #16694:
URL: https://github.com/apache/hudi/issues/16694#issuecomment-3652076294

   @rahil-c @the-other-tim-brown could this issue be addressed by the new type system where the `HoodieSchema` is the single source of truth and there is no more back-and-forth conversion between Spark and Avro schema?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Revisit how we decide the reader schema for reading log files in the filegroup reader [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16694:
URL: https://github.com/apache/hudi/issues/16694#issuecomment-4795632650

   Looks like this has been addressed by the HoodieSchema migration that landed after this issue was filed. PR #17548 (`feat(schema): Migrate log reader and partitioners to take HoodieSchema`, commit `192c3247fd57`) moved `HoodieMergedLogRecordScanner`, `HoodieLogFileReader`, `BaseHoodieLogRecordReader`, and `HoodieAvroDataBlock` over to take `HoodieSchema` directly. `FileGroupReaderSchemaHandler` now exposes `tableSchema` / `requestedSchema` / `requiredSchema` as `HoodieSchema`, and `HoodieSchemaType.ENUM` is a first-class type — so the Spark↔Avro round-trip that turned enums into strings no longer happens on the filegroup reader's log-read path. Suggest closing this (and HUDI-8415) unless there's a specific remaining gap; otherwise it would help to retitle the ticket to point at the concrete sub-case still left.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Revisit how we decide the reader schema for reading log files in the filegroup reader [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16694: Revisit how we decide the reader schema for reading log files in the filegroup reader
URL: https://github.com/apache/hudi/issues/16694


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org