[DISCUSS] Adding Different Storage Support for Fileset in Gravitino

[张涵 <zh...@xiaomi.com.INVALID>]

Hi all,
Gravitino fileset currently supports HDFS as its underlying storage system. In production practice, data may not only be stored on HDFS, so it is necessary to add support for more storage systems like s3, gcs, etc., to expand the capabilities of fileset.
I've opened a Github issue to discuss this proposal: https://github.com/apache/gravitino/issues/4843
And I’ve come up with a draft design document to describe a framework about how to support different types of storage systems: https://docs.google.com/document/d/1qpInooV4MMjE6Be2cJSWPw0fNdDGoUmdJ2AFRyMcoP8/edit?usp=sharing
I'd appreciate if the community has any suggestions or potential issues about the design.
Thanks for your time.
Best,
xiaoz
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

Re: [DISCUSS] Adding Different Storage Support for Fileset in Gravitino

[ShaoFeng Shi <sh...@apache.org>]

Hi Han,

I went through the design, it is clear and easy to understand. Thanks for
drafting that!

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC member,
Apache Incubator PMC member,
Email: shaofengshi@apache.org






张涵 <zh...@xiaomi.com.invalid> 于2024年9月5日周四 18:49写道：

> Hi all,
> Gravitino fileset currently supports HDFS as its underlying storage
> system. In production practice, data may not only be stored on HDFS, so it
> is necessary to add support for more storage systems like s3, gcs, etc., to
> expand the capabilities of fileset.
> I've opened a Github issue to discuss this proposal:
> https://github.com/apache/gravitino/issues/4843
> And I’ve come up with a draft design document to describe a framework
> about how to support different types of storage systems:
> https://docs.google.com/document/d/1qpInooV4MMjE6Be2cJSWPw0fNdDGoUmdJ2AFRyMcoP8/edit?usp=sharing
> I'd appreciate if the community has any suggestions or potential issues
> about the design.
> Thanks for your time.
> Best,
> xiaoz
> #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
> This e-mail and its attachments contain confidential information from
> XIAOMI, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!******/#
>