[jira] [Created] (HUDI-4176) TableSchemaResolver fetches/parses HoodieCommitMetadata multiple times while extracting Schema

["Alexey Kudinkin (Jira)" <ji...@apache.org>]

Alexey Kudinkin created HUDI-4176:
-------------------------------------

             Summary: TableSchemaResolver fetches/parses HoodieCommitMetadata multiple times while extracting Schema
                 Key: HUDI-4176
                 URL: https://issues.apache.org/jira/browse/HUDI-4176
             Project: Apache Hudi
          Issue Type: Bug
    Affects Versions: 0.11.0
            Reporter: Alexey Kudinkin
             Fix For: 0.11.1


We've recently discovered that TableSchemaResolver does a lot of throw-away work during initialization and basic schema reading performed by Spark Datasource (see screenshot).

This poses a problem for large tables where HoodieCommitMetadata is of non-trivial size (100s of Mbs).

We'd minimize amount of throw-away work done by `TableSchemaResolver` and try to re-use read/parsed commits' metadata as much as possible.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)