[SUPPORT] Best Practice for Storing Custom Metadata in Hudi

[Josep Sampé <jo...@qbeast.io>]

Hi everyone,

In our team, we started exploring the integration of Hudi in Qbeast (
qbeast.io), and over the last few months, I've been diving into Hudi's
internals to identify the best place for integration and the most suitable
components to use.

In Qbeast, we’re introducing a new way of indexing data, which requires
additional metadata for each Parquet file. Currently, we’re storing this
metadata in the extraMetadata field of the commit files, as Hudi allows
user-defined metadata there. However, I’m wondering if this is the best
approach or if it would be better to store this information in the metadata
table.

From my understanding:

   - The extraMetadata field is flexible and user-defined, which makes it
   easy to use for custom metadata.
   - The metadata table seems more "closed" and focused on specific
   system-level functionalities.

Would it make more sense to continue using extraMetadata, or is there a
recommended way to extend the metadata table to include custom fields like
these? Any guidance or best practices would be greatly appreciated!

Thanks in advance!

Re: [SUPPORT] Best Practice for Storing Custom Metadata in Hudi

[Sagar Sumit <co...@apache.org>]

Hi Josep,

Thanks for providing more details. Extending the metadata table schema to include your custom metadata is a reasonable approach and aligns with Hudi’s design principles for storing metadata and indexes efficiently.

To extend the schema in the right way, we do need to consider a few things:

1. The structure you mentioned suggests a zonemap-like index, but at a finer granularity - per block within a file. Could you clarify if "blocks" refer to Parquet row groups, or are they specific to Qbeast?

2. Would a more fine-grained column stats index (i.e., extending the column stats payload) suffice, or should we consider introducing a dedicated block-level index, such as a `BlockStatsIndex`? Either way, Hudi’s metadata table supports schema evolution, so introducing new fields would not break existing users.

3. How do you envision updating the block-level stats? The `HoodieBackedTableMetadataWriter` class provides APIs for updating the metadata table, so the primary consideration would be designing the update algorithm for your index.

I would highly encourage you to open an RFC, especially if you believe this index could serve as a general-purpose block-level zonemap and be beneficial to the wider Hudi community. The community would benefit from your insights, particularly if this aligns with broader indexing enhancements in Hudi.

Regards,  
Sagar

On 2025/01/20 16:24:30 Josep Sampé wrote:
> Hi Sivabalan,
> 
> Thanks for your response. The metadata we need to store is indeed per-file,
> and it is leveraged primarily during reads. Currently, we are using the
> extraMetadata field in the commit files, but this approach requires reading
> both the active and archive timelines to extract the information during
> reads.
> 
> We are exploring a solution where the metadata is stored in the
> MetadataTable for faster retrieval and improved performance. This would
> also help align with Hudi's internals, as the MetadataTable is primarily
> used for storing indexes and other metadata-related information.
> 
> In our solution, we would aim to extend the metadata table schema and
> include something like this:
> 
> |-- qbeastMetadata: struct (nullable = true)
>     |-- fileName: string (nullable = false)
>     |-- revision: integer (nullable = false)
>     |-- blocks: struct (nullable = false)
>         |-- id: integer (nullable = false)
>         |-- min: integer (nullable = false)
>         |-- max: integer (nullable = false)
>         |-- elementCount: integer (nullable = false)
> 
> Looking at the code, it seems that the default schema is defined in the
> HoodieMetadata.avsc file (
> https://github.com/apache/hudi/blob/master/hudi-common/src/main/avro/HoodieMetadata.avsc),
> and the classes that manage the table are generated automatically for this
> schema.
> 
> Our question is: what would be the proper way to extend the default schema
> to include the metadata we need and generate the classes to manage it?
> 
> Best regards,
> 
> -Josep
> 

Re: [SUPPORT] Best Practice for Storing Custom Metadata in Hudi

[Josep Sampé <jo...@qbeast.io>]

Hi Sivabalan,

Thanks for your response. The metadata we need to store is indeed per-file,
and it is leveraged primarily during reads. Currently, we are using the
extraMetadata field in the commit files, but this approach requires reading
both the active and archive timelines to extract the information during
reads.

We are exploring a solution where the metadata is stored in the
MetadataTable for faster retrieval and improved performance. This would
also help align with Hudi's internals, as the MetadataTable is primarily
used for storing indexes and other metadata-related information.

In our solution, we would aim to extend the metadata table schema and
include something like this:

|-- qbeastMetadata: struct (nullable = true)
    |-- fileName: string (nullable = false)
    |-- revision: integer (nullable = false)
    |-- blocks: struct (nullable = false)
        |-- id: integer (nullable = false)
        |-- min: integer (nullable = false)
        |-- max: integer (nullable = false)
        |-- elementCount: integer (nullable = false)

Looking at the code, it seems that the default schema is defined in the
HoodieMetadata.avsc file (
https://github.com/apache/hudi/blob/master/hudi-common/src/main/avro/HoodieMetadata.avsc),
and the classes that manage the table are generated automatically for this
schema.

Our question is: what would be the proper way to extend the default schema
to include the metadata we need and generate the classes to manage it?

Best regards,

-Josep

Re: [SUPPORT] Best Practice for Storing Custom Metadata in Hudi

[Sivabalan <n....@gmail.com>]

Hey Josep,
    Thanks for reaching out. Sorry for the delay as it's the holiday season.

Your understanding of metadata table and extraMetadata is right wrt
Hudi. But, can you throw some more light as to how the additional
metadata is leveraged? Are you planning to introduce a new indexing
for writes or for reads? Based on your write up, it is per file
metadata.

By the way, if it is a general purpose index, do you want to develop
it in the open, so that others can benefit as well.

On Thu, 19 Dec 2024 at 11:51, Josep Sampé <jo...@qbeast.io> wrote:
>
> Hi everyone,
>
> In our team, we started exploring the integration of Hudi in Qbeast (
> qbeast.io), and over the last few months, I've been diving into Hudi's
> internals to identify the best place for integration and the most suitable
> components to use.
>
> In Qbeast, we’re introducing a new way of indexing data, which requires
> additional metadata for each Parquet file. Currently, we’re storing this
> metadata in the extraMetadata field of the commit files, as Hudi allows
> user-defined metadata there. However, I’m wondering if this is the best
> approach or if it would be better to store this information in the metadata
> table.
>
> From my understanding:
>
>    - The extraMetadata field is flexible and user-defined, which makes it
>    easy to use for custom metadata.
>    - The metadata table seems more "closed" and focused on specific
>    system-level functionalities.
>
> Would it make more sense to continue using extraMetadata, or is there a
> recommended way to extend the metadata table to include custom fields like
> these? Any guidance or best practices would be greatly appreciated!
>
> Thanks in advance!



-- 
Regards,
-Sivabalan