You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Sivabalan <n....@gmail.com> on 2021/11/05 16:17:26 UTC
[DISCUSS] RFC for Synchronous Metadata table for File listing
RFC-15
<https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements>
made an attempt to boost performance of file listing by storing all file
information in metadata table. As we are looking to build more infra around
metadata table (RFC-27 for data skipping, etc), we felt having a
synchronous design will make it more tighter and will avoid some of the
corner cases with async approach.
So, we will write up a new RFC for file listing based on metadata table
with synchronous updates.
--
Regards,
-Sivabalan
Re: [DISCUSS] RFC for Synchronous Metadata table for File listing
Posted by Manoj Govindassamy <ma...@gmail.com>.
+1 for the synchronous metadata updates. Looking forward to the RFC.
On Fri, Nov 12, 2021 at 4:46 PM Vinoth Chandar <vi...@apache.org> wrote:
> +1 on this.
>
> On Fri, Nov 5, 2021 at 9:17 AM Sivabalan <n....@gmail.com> wrote:
>
> > RFC-15
> > <
> >
> https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements
> > >
> > made an attempt to boost performance of file listing by storing all file
> > information in metadata table. As we are looking to build more infra
> around
> > metadata table (RFC-27 for data skipping, etc), we felt having a
> > synchronous design will make it more tighter and will avoid some of the
> > corner cases with async approach.
> >
> > So, we will write up a new RFC for file listing based on metadata table
> > with synchronous updates.
> >
> > --
> > Regards,
> > -Sivabalan
> >
>
Re: [DISCUSS] RFC for Synchronous Metadata table for File listing
Posted by Vinoth Chandar <vi...@apache.org>.
+1 on this.
On Fri, Nov 5, 2021 at 9:17 AM Sivabalan <n....@gmail.com> wrote:
> RFC-15
> <
> https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements
> >
> made an attempt to boost performance of file listing by storing all file
> information in metadata table. As we are looking to build more infra around
> metadata table (RFC-27 for data skipping, etc), we felt having a
> synchronous design will make it more tighter and will avoid some of the
> corner cases with async approach.
>
> So, we will write up a new RFC for file listing based on metadata table
> with synchronous updates.
>
> --
> Regards,
> -Sivabalan
>