You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Alexey Kudinkin (Jira)" <ji...@apache.org> on 2023/01/13 22:27:00 UTC

[jira] [Created] (HUDI-5556) Make sure HoodieTableMetadata API allows to read both in-memory and on-cluster

Alexey Kudinkin created HUDI-5556:
-------------------------------------

             Summary: Make sure HoodieTableMetadata API allows to read both in-memory and on-cluster
                 Key: HUDI-5556
                 URL: https://issues.apache.org/jira/browse/HUDI-5556
             Project: Apache Hudi
          Issue Type: Improvement
          Components: metadata
            Reporter: Alexey Kudinkin
             Fix For: 0.14.0


Currently, most of the HoodieTableMetadata APIs (getColumnStats, getBloomFilters, etc) provide only for loading the data from MT in-memory which shifts the burden on the caller to break their requests in chunks to make sure these fit in memory: for ex, trying to read Bloom Filters from MT, Bloom Index (caller) have to make sure we're not reading more than 256 filters at a time to limit its memory footprint.

 

Instead HoodieTableMetadata API has to be
 # Rebased to rely on HoodieData
 # Provide levers to the caller whether MT should be read in-memory or on-cluster



--
This message was sent by Atlassian Jira
(v8.20.10#820010)