You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@xtable.apache.org by Vinish Reddy <vi...@apache.org> on 2024/12/06 08:15:42 UTC
[DISCUSS] Sync table formats metadata to multiple catalogs
Hello Apache XTable (Incubating) Community,
This is a discussion regarding a new feature request I have created in GH.
https://github.com/apache/incubator-xtable/issues/590
*Context*
Users of Apache XTable (Incubating) today can translate metadata across
table formats (iceberg, hudi, and delta) and use the tables in different
platforms depending on their choice. Today there's still some friction
involved in terms of usability because users need to explicitly register
the tables <https://xtable.apache.org/docs/catalogs-index/> in the catalog
of their choice (glue, HMS, unity, bigLake etc.) and then use the catalog
in the platform of their choice to do DDL, DML queries.
XTable is built on the principle of omni directional interoperability and
I'm proposing an interface which allows syncing metadata of table formats
to multiple catalogs in a continuous and incremental manner.
*Why do we need this feature ?*1. Reduce friction for XTable users - XTable
sync will register the tables in the catalogs of their choice after
metadata generation. If users are using a single format, they can still use
XTable to sync the metadata across multiple catalogs.
2. Avoid catalog lock-in - There's no reason why data/metadata in storage
should be registered in a single catalog, users can register the table
across multiple catalogs depending on the use-case, ecosystem and features
provided by the catalog.
*Implementation*
I have submitted a PR with the interfaces for CatalogSyncClient and
CatalogSyncOperations,
https://github.com/apache/incubator-xtable/pull/591
Any inputs/feedback from the community who are interested in collaborating
on the design and implementation of this feature to respond to this email
or join the discussion directly on GitHub. Your input, whether in design
suggestions or implementation support, would be appreciated.
Thanks
Vinish
Re: [DISCUSS] Sync table formats metadata to multiple catalogs
Posted by Vinoth Chandar <vi...@apache.org>.
+1 supportive of this as well. We've long held this view in the Apache Hudi
community. XTable can bring those to other table formats as well now.
Curious to hear others' thoughts here..
P.S: should we create a discussion on GH as well - cross link dev thread
for easy discoverability.
On Sat, Dec 7, 2024 at 5:17 PM Kyle Weller <ky...@gmail.com> wrote:
> I also think this is a great idea, I just came from AWS re:Invent
> conference this week and I heard many people bring up this need. I think
> this is also aligned to the project's goals and intentions of bringing
> cross-table interoperability since the metastore or catalog is key to how
> you typically access open table formats. So whether you can sync the
> metadata to 1 or to multiple catalogs, this simple task is sorely needed.
> As more catalogs are starting to grow in the community: Unity Catalog,
> Apache Polaris (incubating), Apache Gravitino, DataHub, etc, the need for
> multiple catalogs may also grow.
>
> Thanks,
> Kyle
>
> On Fri, Dec 6, 2024 at 12:16 AM Vinish Reddy <vi...@apache.org> wrote:
>
> > Hello Apache XTable (Incubating) Community,
> >
> > This is a discussion regarding a new feature request I have created in
> GH.
> > https://github.com/apache/incubator-xtable/issues/590
> >
> > *Context*
> > Users of Apache XTable (Incubating) today can translate metadata across
> > table formats (iceberg, hudi, and delta) and use the tables in different
> > platforms depending on their choice. Today there's still some friction
> > involved in terms of usability because users need to explicitly register
> > the tables <https://xtable.apache.org/docs/catalogs-index/> in the
> catalog
> > of their choice (glue, HMS, unity, bigLake etc.) and then use the catalog
> > in the platform of their choice to do DDL, DML queries.
> >
> > XTable is built on the principle of omni directional interoperability and
> > I'm proposing an interface which allows syncing metadata of table formats
> > to multiple catalogs in a continuous and incremental manner.
> >
> >
> > *Why do we need this feature ?*1. Reduce friction for XTable users -
> XTable
> > sync will register the tables in the catalogs of their choice after
> > metadata generation. If users are using a single format, they can still
> use
> > XTable to sync the metadata across multiple catalogs.
> > 2. Avoid catalog lock-in - There's no reason why data/metadata in storage
> > should be registered in a single catalog, users can register the table
> > across multiple catalogs depending on the use-case, ecosystem and
> features
> > provided by the catalog.
> >
> > *Implementation*
> > I have submitted a PR with the interfaces for CatalogSyncClient and
> > CatalogSyncOperations,
> > https://github.com/apache/incubator-xtable/pull/591
> >
> > Any inputs/feedback from the community who are interested in
> collaborating
> > on the design and implementation of this feature to respond to this email
> > or join the discussion directly on GitHub. Your input, whether in design
> > suggestions or implementation support, would be appreciated.
> >
> > Thanks
> > Vinish
> >
>
Re: [DISCUSS] Sync table formats metadata to multiple catalogs
Posted by Kyle Weller <ky...@gmail.com>.
I also think this is a great idea, I just came from AWS re:Invent
conference this week and I heard many people bring up this need. I think
this is also aligned to the project's goals and intentions of bringing
cross-table interoperability since the metastore or catalog is key to how
you typically access open table formats. So whether you can sync the
metadata to 1 or to multiple catalogs, this simple task is sorely needed.
As more catalogs are starting to grow in the community: Unity Catalog,
Apache Polaris (incubating), Apache Gravitino, DataHub, etc, the need for
multiple catalogs may also grow.
Thanks,
Kyle
On Fri, Dec 6, 2024 at 12:16 AM Vinish Reddy <vi...@apache.org> wrote:
> Hello Apache XTable (Incubating) Community,
>
> This is a discussion regarding a new feature request I have created in GH.
> https://github.com/apache/incubator-xtable/issues/590
>
> *Context*
> Users of Apache XTable (Incubating) today can translate metadata across
> table formats (iceberg, hudi, and delta) and use the tables in different
> platforms depending on their choice. Today there's still some friction
> involved in terms of usability because users need to explicitly register
> the tables <https://xtable.apache.org/docs/catalogs-index/> in the catalog
> of their choice (glue, HMS, unity, bigLake etc.) and then use the catalog
> in the platform of their choice to do DDL, DML queries.
>
> XTable is built on the principle of omni directional interoperability and
> I'm proposing an interface which allows syncing metadata of table formats
> to multiple catalogs in a continuous and incremental manner.
>
>
> *Why do we need this feature ?*1. Reduce friction for XTable users - XTable
> sync will register the tables in the catalogs of their choice after
> metadata generation. If users are using a single format, they can still use
> XTable to sync the metadata across multiple catalogs.
> 2. Avoid catalog lock-in - There's no reason why data/metadata in storage
> should be registered in a single catalog, users can register the table
> across multiple catalogs depending on the use-case, ecosystem and features
> provided by the catalog.
>
> *Implementation*
> I have submitted a PR with the interfaces for CatalogSyncClient and
> CatalogSyncOperations,
> https://github.com/apache/incubator-xtable/pull/591
>
> Any inputs/feedback from the community who are interested in collaborating
> on the design and implementation of this feature to respond to this email
> or join the discussion directly on GitHub. Your input, whether in design
> suggestions or implementation support, would be appreciated.
>
> Thanks
> Vinish
>