You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/01/09 12:07:11 UTC
[I] Variant Blog Post [parquet-site]
alamb opened a new issue, #147:
URL: https://github.com/apache/parquet-site/issues/147
- Part of https://github.com/apache/parquet-site/issues/146
We recently added Variant to parquet format -- see https://parquet.apache.org/docs/file-format/types/variantencoding/
However, the only documentation that currently exists is the low level technical spec.
This is somewhat akward to explain to others how we should add Variant support to various systems: there is no high level overview / link to point people at.
I suggest a blog post with a a high level
1. Explains the usecase of Variant (semi-structured data)
2. Gives a technical overview of the encoding (with diagrams)
3. Explains how shredding works, gives some examples (with diagrams)
I would love a story arc that also highlights the fact that this is a major new addition to the Parquet spec and that it has already seen wide adoption https://parquet.apache.org/docs/file-format/implementationstatus/
Some existing material we should feel free to re-use:
1. [slides from Accelerating Apache Parquet with metadata stores and specialized indexes using Apache DataFusion ](https://docs.google.com/presentation/d/1e_Z_F8nt2rcvlNvhU11khF5lzJJVqNtqtyJ-G3mp4-Q), [recording ](https://www.youtube.com/watch?v=74YsJT1-Rdk)YouTube
2. [Introduction to Iceberg / Parquet Variant NYC Apache Iceberg™ Community Meetup](https://docs.google.com/presentation/d/1NN583KuJ3nelIrrH64HAASmbRtO9khhNSO3fw0_Nslc)
3. Original databricks variant announcement: https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark
4. DataBricks announcement: https://www.databricks.com/blog/introducing-variant-new-open-standard-semi-structured-data-apache-parquettm-delta-lake (this focuses quite a bit on system-level integration of variant in DataBricks and reads more like product feature announcement rather than a technical overview)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Blog on Variant [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #147:
URL: https://github.com/apache/parquet-site/issues/147#issuecomment-3890783945
I think the draft looks great (I took another pass through the google doc)
The next step I suggest is to turn it into markdown, perhaps following this PR as an example
- https://github.com/apache/parquet-site/pull/156
I can help if that would be useful, otherwise feel free to put up a PR yourself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Blog on Variant [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #147:
URL: https://github.com/apache/parquet-site/issues/147#issuecomment-3728720818
I believe @aihuaxu said on the mailing list https://lists.apache.org/thread/278m3oyvg3okl819xo55m4ttogbzbjlc that he was working on one
> Agree. We started to write a little bit for variant and I can take variant blog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Blog on Variant [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #147:
URL: https://github.com/apache/parquet-site/issues/147#issuecomment-3738529904
> > I believe [@aihuaxu](https://github.com/aihuaxu) said on the mailing list https://lists.apache.org/thread/278m3oyvg3okl819xo55m4ttogbzbjlc that he was working on one
> > > Agree. We started to write a little bit for variant and I can take variant blog.
>
> [@alamb](https://github.com/alamb) We have published one here https://www.snowflake.com/en/engineering-blog/apache-iceberg-v3-variant-type/. I guess we cannot directly reuse it and I can work on top of it for Parquet blog?
Thanks @aihuaxu -- that blog is a nice read. I think much of it could be re-used for a Parquet blog (with you and Snowflake's permission of course)
However I think the fact the blog also talks about snowflake makes it a bit unclear where the boundary between the two items are and I think the blog (understandably) seems to have an intended audience of "end users" of Variant
For a parquet blog I was envisioning something which:
1. Is focused on parquet
1. Target audience is people implementing Variant (and those who are interested in such a level of detail)
2. Has some examples of encodings (maybe we can use the example shredding from the snowflake blog)
3. Also emphasizes that this is a major addition to Parquet that has seen rapid uptake across its ecosystem (Java, C++, Rust, Go, DuckDB, etc), as an example of how the format continues to evolve.
(to be clear, I am not saying you have to write such a blog, I am just articulating what I think would be most valuable for Parquet)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Blog on Variant [parquet-site]
Posted by "aihuaxu (via GitHub)" <gi...@apache.org>.
aihuaxu commented on issue #147:
URL: https://github.com/apache/parquet-site/issues/147#issuecomment-3729712094
> I believe [@aihuaxu](https://github.com/aihuaxu) said on the mailing list https://lists.apache.org/thread/278m3oyvg3okl819xo55m4ttogbzbjlc that he was working on one
>
> > Agree. We started to write a little bit for variant and I can take variant blog.
@alamb We have published one here https://www.snowflake.com/en/engineering-blog/apache-iceberg-v3-variant-type/. I guess we cannot directly reuse it and I can work on top of it for Parquet blog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Blog on Variant [parquet-site]
Posted by "aihuaxu (via GitHub)" <gi...@apache.org>.
aihuaxu commented on issue #147:
URL: https://github.com/apache/parquet-site/issues/147#issuecomment-3831800365
I have drafted blog for variant (https://docs.google.com/document/d/1ABr3p-xj_8rHQ2kdzzDSceejGkU0nWnriZGPjhDoDBc/edit?tab=t.0). Please help review or let me know if you want to collaborate on the doc. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Blog on Variant [parquet-site]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #147:
URL: https://github.com/apache/parquet-site/issues/147#issuecomment-3842760187
I reviewed the draft and left some feedback. TLDR is it looks great to me -- thanks @aihuaxu 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Blog on Variant [parquet-site]
Posted by "aihuaxu (via GitHub)" <gi...@apache.org>.
aihuaxu commented on issue #147:
URL: https://github.com/apache/parquet-site/issues/147#issuecomment-3739570110
> > > I believe [@aihuaxu](https://github.com/aihuaxu) said on the mailing list https://lists.apache.org/thread/278m3oyvg3okl819xo55m4ttogbzbjlc that he was working on one
> > > > Agree. We started to write a little bit for variant and I can take variant blog.
> >
> >
> > [@alamb](https://github.com/alamb) We have published one here https://www.snowflake.com/en/engineering-blog/apache-iceberg-v3-variant-type/. I guess we cannot directly reuse it and I can work on top of it for Parquet blog?
>
> Thanks [@aihuaxu](https://github.com/aihuaxu) -- that blog is a nice read. I think much of it could be re-used for a Parquet blog (with you and Snowflake's permission of course)
>
> However I think the fact the blog also talks about Iceberg makes it a bit unclear where the boundary between the two items are and I think the blog (understandably) seems to have an intended audience of "end users" of Variant
>
> For a parquet blog I was envisioning something which:
>
> 1. Is focused on parquet
> 2. Target audience is people implementing Variant (and those who are interested in such a level of detail)
> 3. Has some examples of encodings (maybe we can use the example shredding from the snowflake blog)
> 4. Also emphasizes that this is a major addition to Parquet that has seen rapid uptake across its ecosystem (Java, C++, Rust, Go, DuckDB, etc), as an example of how the format continues to evolve.
>
> (to be clear, I am not saying you have to write such a blog, I am just articulating what I think would be most valuable for Parquet)
That's right. The current one is more intended for end user. Thanks for the feedback and that makes sense to me. Let me draft one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org