You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gravitino.apache.org by Minghuang Li <mc...@apache.org> on 2024/08/15 08:26:02 UTC

[DISCUSS] Adding Apache Hudi Catalog Support in Gravitino

Hi Y’all,

Gravitino currently supports the Apache Iceberg catalog[1] and Apache Paimon catalog[2] for metadata management but does not support Apache Hudi[3] yet. I'd like to start a discussion about adding Apache Hudi catalog support to Gravitino. 

This integration could potentially expand Gravitino's capabilities in managing data lake metadata.

I've opened a Github epic issue to track this proposal: #4306<https://github.com/apache/gravitino/issues/4306>

A draft design document is available here (will also be posted to issue): <https://docs.google.com/document/d/1kJN4JQaQ_-_rdk9jR4fiL-LUxqRaSG00q9cSmqWBNR8/edit?usp=sharing>

I'd appreciate the community's thoughts on:
1. Is this feature aligned with Gravitino's goals?
2. Are there any concerns or potential issues we should address?
3. Any suggestions for improving the design or implementation approach?

If there's interest in pursuing this, I'm willing to move forward as outlined in the timeline. However, I'm open to any feedback or alternative approaches the community might suggest.

Thanks for your time and input.

Best regards,
Minghuang Li

[1] https://github.com/apache/gravitino/blob/main/docs/lakehouse-iceberg-catalog.md
[2] https://github.com/apache/gravitino/blob/main/docs/lakehouse-paimon-catalog.md
[3] https://hudi.apache.org

Re: [DISCUSS] Adding Apache Hudi Catalog Support in Gravitino

Posted by Nicholas Jiang <ni...@apache.org>.

Thanks Minghuang for driving Apache Hudi Catalog Support Proposal.

IMO, you could use metadata synchronization tool to support Hudi Catalog based on Hudi 0.15 version, which version supports Spark 3.3, 3.4 and 3.5. Meanwhile, you can also start discussion thread about providing a unified Catalog in Apache Hudi community. How to ensure compatibility between Hudi versions also needs to be covered in the proposal.

Forward to Apache Hudi Catalog support.

Regards,
Nicholas Jiang 

On 2024/08/15 08:26:02 Minghuang Li wrote:
> Hi Y’all,
> 
> Gravitino currently supports the Apache Iceberg catalog[1] and Apache Paimon catalog[2] for metadata management but does not support Apache Hudi[3] yet. I'd like to start a discussion about adding Apache Hudi catalog support to Gravitino. 
> 
> This integration could potentially expand Gravitino's capabilities in managing data lake metadata.
> 
> I've opened a Github epic issue to track this proposal: #4306<https://github.com/apache/gravitino/issues/4306>
> 
> A draft design document is available here (will also be posted to issue): <https://docs.google.com/document/d/1kJN4JQaQ_-_rdk9jR4fiL-LUxqRaSG00q9cSmqWBNR8/edit?usp=sharing>
> 
> I'd appreciate the community's thoughts on:
> 1. Is this feature aligned with Gravitino's goals?
> 2. Are there any concerns or potential issues we should address?
> 3. Any suggestions for improving the design or implementation approach?
> 
> If there's interest in pursuing this, I'm willing to move forward as outlined in the timeline. However, I'm open to any feedback or alternative approaches the community might suggest.
> 
> Thanks for your time and input.
> 
> Best regards,
> Minghuang Li
> 
> [1] https://github.com/apache/gravitino/blob/main/docs/lakehouse-iceberg-catalog.md
> [2] https://github.com/apache/gravitino/blob/main/docs/lakehouse-paimon-catalog.md
> [3] https://hudi.apache.org
>

Re: [DISCUSS] Adding Apache Hudi Catalog Support in Gravitino

Posted by Nicholas Jiang <ni...@apache.org>.

Thanks Minghuang for driving Apache Hudi Catalog Support Proposal.

IMO, you could use metadata synchronization tool to support Hudi Catalog based on Hudi 0.15 version, which version supports Spark 3.3, 3.4 and 3.5. Meanwhile, you can also start discussion thread about providing a unified Catalog in Apache Hudi community. How to ensure compatibility between Hudi versions also needs to be covered in the proposal.

Forward to Apache Hudi Catalog support.

Regards,
Nicholas Jiang 

On 2024/08/15 08:26:02 Minghuang Li wrote:
> Hi Y’all,
> 
> Gravitino currently supports the Apache Iceberg catalog[1] and Apache Paimon catalog[2] for metadata management but does not support Apache Hudi[3] yet. I'd like to start a discussion about adding Apache Hudi catalog support to Gravitino. 
> 
> This integration could potentially expand Gravitino's capabilities in managing data lake metadata.
> 
> I've opened a Github epic issue to track this proposal: #4306<https://github.com/apache/gravitino/issues/4306>
> 
> A draft design document is available here (will also be posted to issue): <https://docs.google.com/document/d/1kJN4JQaQ_-_rdk9jR4fiL-LUxqRaSG00q9cSmqWBNR8/edit?usp=sharing>
> 
> I'd appreciate the community's thoughts on:
> 1. Is this feature aligned with Gravitino's goals?
> 2. Are there any concerns or potential issues we should address?
> 3. Any suggestions for improving the design or implementation approach?
> 
> If there's interest in pursuing this, I'm willing to move forward as outlined in the timeline. However, I'm open to any feedback or alternative approaches the community might suggest.
> 
> Thanks for your time and input.
> 
> Best regards,
> Minghuang Li
> 
> [1] https://github.com/apache/gravitino/blob/main/docs/lakehouse-iceberg-catalog.md
> [2] https://github.com/apache/gravitino/blob/main/docs/lakehouse-paimon-catalog.md
> [3] https://hudi.apache.org
>

Re: [DISCUSS] Adding Apache Hudi Catalog Support in Gravitino

Posted by Minghuang Li <mc...@apache.org>.

Hello Rory and Nicholas,

Thank you both for your valuable suggestions! Following your advice, I have already sent an inquiry email<https://lists.apache.org/thread/xpdgx619dodp64h3bgo2m9cfkrbdy6d6> to the Hudi dev community. Additionally, I will proceed with our planned implementation of the Hudi catalog while waiting for feedback from the community.

Initially, I will work on developing a read-only HMS backend for the Hudi catalog based on Hudi version 0.15.0, adhering to Gravitino's interface specifications. During this phase, I will carefully abstract the integration to minimize the impact of potential future changes from the Hudi community, which might otherwise require extensive refactoring.

Further updates on the progress of the Hudi catalog will be tracked on GitHub issue<https://github.com/apache/gravitino/issues/4306>.

Best regards,
Minghuang Li

On 2024/08/19 09:03:11 roryqi wrote:
> Thanks Minghuang for bringing out this.  Maybe you can bring this
> discussion to the Hui dev mail list.
> 
> Rory
> 
> Jerry Shao <js...@apache.org> 于2024年8月16日周五 19:45写道：
> 
> > Thanks Minghuang for bringing out this. I will take a look at the details
> > and get back to you soon.
> >
> > Best,
> > Jerry
> >
> > Minghuang Li <mc...@apache.org> 于2024年8月15日周四 16:26写道：
> >
> > > Hi Y’all,
> > >
> > > Gravitino currently supports the Apache Iceberg catalog[1] and Apache
> > > Paimon catalog[2] for metadata management but does not support Apache
> > > Hudi[3] yet. I'd like to start a discussion about adding Apache Hudi
> > > catalog support to Gravitino.
> > >
> > > This integration could potentially expand Gravitino's capabilities in
> > > managing data lake metadata.
> > >
> > > I've opened a Github epic issue to track this proposal: #4306<
> > > https://github.com/apache/gravitino/issues/4306>
> > >
> > > A draft design document is available here (will also be posted to
> > issue): <
> > >
> > https://docs.google.com/document/d/1kJN4JQaQ_-_rdk9jR4fiL-LUxqRaSG00q9cSmqWBNR8/edit?usp=sharing
> > > >
> > >
> > > I'd appreciate the community's thoughts on:
> > > 1. Is this feature aligned with Gravitino's goals?
> > > 2. Are there any concerns or potential issues we should address?
> > > 3. Any suggestions for improving the design or implementation approach?
> > >
> > > If there's interest in pursuing this, I'm willing to move forward as
> > > outlined in the timeline. However, I'm open to any feedback or
> > alternative
> > > approaches the community might suggest.
> > >
> > > Thanks for your time and input.
> > >
> > > Best regards,
> > > Minghuang Li
> > >
> > > [1]
> > >
> > https://github.com/apache/gravitino/blob/main/docs/lakehouse-iceberg-catalog.md
> > > [2]
> > >
> > https://github.com/apache/gravitino/blob/main/docs/lakehouse-paimon-catalog.md
> > > [3] https://hudi.apache.org
> > >
> >
>

Re: [DISCUSS] Adding Apache Hudi Catalog Support in Gravitino

Posted by roryqi <ro...@apache.org>.

Thanks Minghuang for bringing out this.  Maybe you can bring this
discussion to the Hui dev mail list.

Rory

Jerry Shao <js...@apache.org> 于2024年8月16日周五 19:45写道：

> Thanks Minghuang for bringing out this. I will take a look at the details
> and get back to you soon.
>
> Best,
> Jerry
>
> Minghuang Li <mc...@apache.org> 于2024年8月15日周四 16:26写道：
>
> > Hi Y’all,
> >
> > Gravitino currently supports the Apache Iceberg catalog[1] and Apache
> > Paimon catalog[2] for metadata management but does not support Apache
> > Hudi[3] yet. I'd like to start a discussion about adding Apache Hudi
> > catalog support to Gravitino.
> >
> > This integration could potentially expand Gravitino's capabilities in
> > managing data lake metadata.
> >
> > I've opened a Github epic issue to track this proposal: #4306<
> > https://github.com/apache/gravitino/issues/4306>
> >
> > A draft design document is available here (will also be posted to
> issue): <
> >
> https://docs.google.com/document/d/1kJN4JQaQ_-_rdk9jR4fiL-LUxqRaSG00q9cSmqWBNR8/edit?usp=sharing
> > >
> >
> > I'd appreciate the community's thoughts on:
> > 1. Is this feature aligned with Gravitino's goals?
> > 2. Are there any concerns or potential issues we should address?
> > 3. Any suggestions for improving the design or implementation approach?
> >
> > If there's interest in pursuing this, I'm willing to move forward as
> > outlined in the timeline. However, I'm open to any feedback or
> alternative
> > approaches the community might suggest.
> >
> > Thanks for your time and input.
> >
> > Best regards,
> > Minghuang Li
> >
> > [1]
> >
> https://github.com/apache/gravitino/blob/main/docs/lakehouse-iceberg-catalog.md
> > [2]
> >
> https://github.com/apache/gravitino/blob/main/docs/lakehouse-paimon-catalog.md
> > [3] https://hudi.apache.org
> >
>

Re: [DISCUSS] Adding Apache Hudi Catalog Support in Gravitino

Posted by Jerry Shao <js...@apache.org>.

Thanks Minghuang for bringing out this. I will take a look at the details
and get back to you soon.

Best,
Jerry

Minghuang Li <mc...@apache.org> 于2024年8月15日周四 16:26写道：

> Hi Y’all,
>
> Gravitino currently supports the Apache Iceberg catalog[1] and Apache
> Paimon catalog[2] for metadata management but does not support Apache
> Hudi[3] yet. I'd like to start a discussion about adding Apache Hudi
> catalog support to Gravitino.
>
> This integration could potentially expand Gravitino's capabilities in
> managing data lake metadata.
>
> I've opened a Github epic issue to track this proposal: #4306<
> https://github.com/apache/gravitino/issues/4306>
>
> A draft design document is available here (will also be posted to issue): <
> https://docs.google.com/document/d/1kJN4JQaQ_-_rdk9jR4fiL-LUxqRaSG00q9cSmqWBNR8/edit?usp=sharing
> >
>
> I'd appreciate the community's thoughts on:
> 1. Is this feature aligned with Gravitino's goals?
> 2. Are there any concerns or potential issues we should address?
> 3. Any suggestions for improving the design or implementation approach?
>
> If there's interest in pursuing this, I'm willing to move forward as
> outlined in the timeline. However, I'm open to any feedback or alternative
> approaches the community might suggest.
>
> Thanks for your time and input.
>
> Best regards,
> Minghuang Li
>
> [1]
> https://github.com/apache/gravitino/blob/main/docs/lakehouse-iceberg-catalog.md
> [2]
> https://github.com/apache/gravitino/blob/main/docs/lakehouse-paimon-catalog.md
> [3] https://hudi.apache.org
>