You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Parth Chandra <pa...@apache.org> on 2017/12/08 17:37:23 UTC
Parquet's new table format
FYI
The Parquet is working on introducing new table format called 'Iceberg' [1]
that has interesting and useful features.
Take a look at the initial post.
[1]
https://lists.apache.org/thread.html/f90ac1c268dea4077e358df1df8dd48f3766db8d4db476c3e0d9baa8@%3Cdev.parquet.apache.org%3E
Re: Parquet's new table format
Posted by Joel Pfaff <jo...@gmail.com>.
Hello,
Iceberg's code has been pushed yesterday to github:
https://github.com/Netflix/iceberg
And the draft of specs have been published there:
https://docs.google.com/document/d/1Q-zL5lSCle6NEEdyfiYsXYzX_
Q8Qf0ctMyGBKslOswA/edit?usp=sharing
--
Joel
On Sat, Dec 9, 2017 at 4:18 AM, Paul Rogers <pr...@mapr.com> wrote:
> Very cool indeed.
>
> Iceberg, if I understand the post, is a file container format: it
> identifies the set of files (in this case, Parquet files) that make up a
> “table.” Since Iceberg mentions Hive, it presumably would work for any file
> format (since it is just a file container.)
>
> This would be a great way to solve our Parquet metadata problem: it allows
> us to identify the set of files that define Drill’s Parquet metadata, and
> to make changes to that metadata in a transactional way.
>
> I wonder if Iceberg could be augmented to include both data and metadata
> in the same container? That way, Drill views and/or Parquet metadata could
> be managed as a unit with the files they describe.
>
> Further, it would seem that Iceberg might be extended to support MVCC by
> listing the files that make up each version (or, equivalently, by listing
> the deltas between versions.)
>
> This is definitely something to watch. Thanks, Parth for bringing it to
> our attention (and thanks to Netflix for open sourcing the format).
>
> - Paul
>
> > On Dec 8, 2017, at 9:37 AM, Parth Chandra <pa...@apache.org> wrote:
> >
> > FYI
> >
> > The Parquet is working on introducing new table format called 'Iceberg'
> [1]
> > that has interesting and useful features.
> >
> > Take a look at the initial post.
> >
> >
> > [1]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.
> apache.org_thread.html_f90ac1c268dea4077e358df1df8dd4
> 8f3766db8d4db476c3e0d9baa8-40-253Cdev.parquet.apache.org-253E&d=DwIBaQ&c=
> cskdkSMqhcnjZxdQVpwTXg&r=Dz59a-Un_5n3KbQ2RYN0KA&m=PHcXxx7w-DDoW_UM90WJzI_
> LGQAhQAGqF1z-3z9xIRc&s=NkV7as3K7vfi7rObvLdSmeDv58FdehfcqxhfRqSBjTU&e=
>
>
Re: Parquet's new table format
Posted by Paul Rogers <pr...@mapr.com>.
Very cool indeed.
Iceberg, if I understand the post, is a file container format: it identifies the set of files (in this case, Parquet files) that make up a “table.” Since Iceberg mentions Hive, it presumably would work for any file format (since it is just a file container.)
This would be a great way to solve our Parquet metadata problem: it allows us to identify the set of files that define Drill’s Parquet metadata, and to make changes to that metadata in a transactional way.
I wonder if Iceberg could be augmented to include both data and metadata in the same container? That way, Drill views and/or Parquet metadata could be managed as a unit with the files they describe.
Further, it would seem that Iceberg might be extended to support MVCC by listing the files that make up each version (or, equivalently, by listing the deltas between versions.)
This is definitely something to watch. Thanks, Parth for bringing it to our attention (and thanks to Netflix for open sourcing the format).
- Paul
> On Dec 8, 2017, at 9:37 AM, Parth Chandra <pa...@apache.org> wrote:
>
> FYI
>
> The Parquet is working on introducing new table format called 'Iceberg' [1]
> that has interesting and useful features.
>
> Take a look at the initial post.
>
>
> [1]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_f90ac1c268dea4077e358df1df8dd48f3766db8d4db476c3e0d9baa8-40-253Cdev.parquet.apache.org-253E&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=Dz59a-Un_5n3KbQ2RYN0KA&m=PHcXxx7w-DDoW_UM90WJzI_LGQAhQAGqF1z-3z9xIRc&s=NkV7as3K7vfi7rObvLdSmeDv58FdehfcqxhfRqSBjTU&e=