You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Sagar Sumit (Jira)" <ji...@apache.org> on 2024/04/10 05:57:00 UTC
[jira] [Closed] (HUDI-7583) Read log block header only for the schema and instant time
[ https://issues.apache.org/jira/browse/HUDI-7583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sagar Sumit closed HUDI-7583.
-----------------------------
Resolution: Fixed
> Read log block header only for the schema and instant time
> ----------------------------------------------------------
>
> Key: HUDI-7583
> URL: https://issues.apache.org/jira/browse/HUDI-7583
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.15.0, 1.0.0
>
>
> The TableSchemaResolver reads the schema from the log file header. The current way of instantiating log reader does not lazily read the content, causing the whole block content to be read, which is unnecessary. This causes the OOM on the Spark driver during clustering when clustering rewrites a file group that contains log files, which requires deriving the schema from the file group in the current logic.
> {code:java}
> public static MessageType readSchemaFromLogFile(FileSystem fs, Path path) throws IOException {
> try (Reader reader = HoodieLogFormat.newReader(fs, new HoodieLogFile(path), null)) {
> HoodieDataBlock lastBlock = null;
> while (reader.hasNext()) {
> HoodieLogBlock block = reader.next();
> if (block instanceof HoodieDataBlock) {
> lastBlock = (HoodieDataBlock) block;
> }
> }
> return lastBlock != null ? new AvroSchemaConverter().convert(lastBlock.getSchema()) : null;
> }
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)