You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:55:26 UTC
[I] Use HoodieHiveCatalog to infer by read log file will exit due to oom [hudi]
hudi-bot opened a new issue, #15468:
URL: https://github.com/apache/hudi/issues/15468
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4950
- Type: Bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use HoodieHiveCatalog to infer by read log file will exit due to oom [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15468:
URL: https://github.com/apache/hudi/issues/15468#issuecomment-4795431019
On current master, the schema-inference path that `HoodieHiveCatalog.getTable` falls into (`StreamerUtil.getLatestTableSchema` → `TableSchemaResolver.readSchemaFromLogFile`) is no longer expected to OOM:
- The schema is read from the **log block header** (`HoodieDataBlock.getSchema()` returns the schema parsed from `HeaderMetadataType.SCHEMA`), not from block content.
- For all log format versions ≥ 1 (the format used by modern Hudi tables), `HoodieLogBlock.tryReadContent` **skips** the block content (seeks past it) instead of loading the bytes — see `HoodieLogFileReader.readBlock` where `shouldReadLazily = nextBlockVersion.getVersion() != HoodieLogFormatVersion.DEFAULT_VERSION`.
- The iteration in `readSchemaFromLogFile` overwrites `lastBlock` each step, so previously read blocks become GC-eligible; memory stays bounded regardless of log file size.
- `fetchSchemaFromFiles` uses `findFirst()` to short-circuit after the first non-null schema, so it does not scan all log files.
The lazy-read behavior was solidified in HUDI-7584 (`ab0e2cdd579b`, Apr 2024), which removed the `readBlockLazily` toggle and made lazy content skipping the only path for v1+ log blocks. Closing as resolved unless a reproducer on current master shows otherwise — please share the JVM heap, table layout, and a stack trace if the OOM is still reachable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use HoodieHiveCatalog to infer by read log file will exit due to oom [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15468:
URL: https://github.com/apache/hudi/issues/15468#issuecomment-3630145999
### Linked PR(s)
- https://github.com/apache/hudi/pull/6827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use HoodieHiveCatalog to infer by read log file will exit due to oom [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15468: Use HoodieHiveCatalog to infer by read log file will exit due to oom
URL: https://github.com/apache/hudi/issues/15468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org