You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Matt Topol <zo...@gmail.com> on 2025/05/08 22:00:56 UTC

[DISCUSS] Arrow Variant Extension Type

Hey All,

There's been various discussions occurring on many different thread
locations (issues, PRs, and so on)[1][2][3], and more that I haven't
linked to, concerning what a canonical Variant Extension Type for
Arrow might look like. As I've looked into implementing some things,
I've also spoken with members of the Arrow, Iceberg and Parquet
communities as to what a good representation for Arrow Variant would
be like in order to ensure good support and adoption.

I also looked at the ClickHouse variant implementation [4]. The
ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
so we don't need to do any extra work there to support it.

So, after discussions and looking into the needs for engines and so
on, I've iterated and written up a proposal for what a Canonical
Variant Extension Type for Arrow could be in a google doc[5]. I'm
hoping that this can spark some discussion and comments on the
document. If there's relative consensus on it, then I'll work on
creating some implementations of it that I can use to formally propose
the addition to the Canonical Extensions.

Please take a read and leave comments on the google doc or on this
thread. Thanks everyone!

--Matt

[1]: https://github.com/apache/arrow-rs/issues/7063
[2]: https://github.com/apache/arrow/issues/45937
[3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
[4]: https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
[5]: https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Dewey Dunnington <de...@gmail.com>.
Thanks for pursuing this! This is one of the questions I had reviewing the
beginnings of Variant support in Parquet C++ and Go. I would love to see
this as a canonical extension type, both for its utility as a type and as
further incentive to strengthen the extension system.

Cheers,

-dewey

On Thu, May 8, 2025 at 9:05 PM Ian Cook <ia...@apache.org> wrote:

> As Parquet adds new types including Variant, I think it's important
> for us to (a) preserve the ability to efficiently round-trip the full
> set of types between Arrow and Parquet, and (b) add new Arrow types
> (or canonical extension types) to enable Arrow to continue to flourish
> in its role as the in-memory and on-the-wire counterpart to Parquet
> storage. Thank you Matt for pursuing this.
>
> Ian
>
>
> On Thu, May 8, 2025 at 6:03 PM Matt Topol <zo...@gmail.com> wrote:
> >
> > Hey All,
> >
> > There's been various discussions occurring on many different thread
> > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > linked to, concerning what a canonical Variant Extension Type for
> > Arrow might look like. As I've looked into implementing some things,
> > I've also spoken with members of the Arrow, Iceberg and Parquet
> > communities as to what a good representation for Arrow Variant would
> > be like in order to ensure good support and adoption.
> >
> > I also looked at the ClickHouse variant implementation [4]. The
> > ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> > so we don't need to do any extra work there to support it.
> >
> > So, after discussions and looking into the needs for engines and so
> > on, I've iterated and written up a proposal for what a Canonical
> > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > hoping that this can spark some discussion and comments on the
> > document. If there's relative consensus on it, then I'll work on
> > creating some implementations of it that I can use to formally propose
> > the addition to the Canonical Extensions.
> >
> > Please take a read and leave comments on the google doc or on this
> > thread. Thanks everyone!
> >
> > --Matt
> >
> > [1]: https://github.com/apache/arrow-rs/issues/7063
> > [2]: https://github.com/apache/arrow/issues/45937
> > [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > [4]:
> https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > [5]:
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Ian Cook <ia...@apache.org>.
As Parquet adds new types including Variant, I think it's important
for us to (a) preserve the ability to efficiently round-trip the full
set of types between Arrow and Parquet, and (b) add new Arrow types
(or canonical extension types) to enable Arrow to continue to flourish
in its role as the in-memory and on-the-wire counterpart to Parquet
storage. Thank you Matt for pursuing this.

Ian


On Thu, May 8, 2025 at 6:03 PM Matt Topol <zo...@gmail.com> wrote:
>
> Hey All,
>
> There's been various discussions occurring on many different thread
> locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> linked to, concerning what a canonical Variant Extension Type for
> Arrow might look like. As I've looked into implementing some things,
> I've also spoken with members of the Arrow, Iceberg and Parquet
> communities as to what a good representation for Arrow Variant would
> be like in order to ensure good support and adoption.
>
> I also looked at the ClickHouse variant implementation [4]. The
> ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> so we don't need to do any extra work there to support it.
>
> So, after discussions and looking into the needs for engines and so
> on, I've iterated and written up a proposal for what a Canonical
> Variant Extension Type for Arrow could be in a google doc[5]. I'm
> hoping that this can spark some discussion and comments on the
> document. If there's relative consensus on it, then I'll work on
> creating some implementations of it that I can use to formally propose
> the addition to the Canonical Extensions.
>
> Please take a read and leave comments on the google doc or on this
> thread. Thanks everyone!
>
> --Matt
>
> [1]: https://github.com/apache/arrow-rs/issues/7063
> [2]: https://github.com/apache/arrow/issues/45937
> [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> [4]: https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> [5]: https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Antoine Pitrou <an...@python.org>.
Le 12/05/2025 à 18:20, Matt Topol a écrit :
 > It's not just Parquet Variant, it's also Iceberg (which has
 > standardized on this) and Spark in-memory (where this encoding scheme
 > originated).

Ok, but it's called Parquet Variant now, since that's where the binary 
spec lives:
https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
https://github.com/apache/parquet-format/blob/master/VariantShredding.md

> As far as relying on union types, the reason we can't do so is because
> the specific purpose of this Variant type is that we don't know the
> types up front, it's dynamic. Also Arrow union types do not allow for> arbitrarily recursive structures, it's one of the issues that others
> have run into with the Arrow union types. Another consideration is the
> goal of projecting the shredded columns. Since we need to project
> shredded columns, we'd only be able to use Sparse Unions, not dense
> ones, and while it is well defined how to project the child of a
> struct it's less defined how to project the child of a union.

Those are good points, thank you.

Regards

Antoine.


Re: [DISCUSS] Arrow Variant Extension Type

Posted by Matt Topol <zo...@gmail.com>.
@Andrew: The current proposal in the document is essentially zero-copy
compatible with the Parquet Variant (give or take the differences in
how variable length binary values are represented in Parquet vs Arrow,
and the proposal allows for  any of Binary/LargeBinary/BinaryView to
represent the values).

@everyone: So far it seems we've hit most of a consensus on the doc,
so if no one has any further significant changes to request/suggest
then I'm going to start creating an implementation of the Variant
Extension type. Since we need implementations to have a vote for it to
become a canonical extension type.

--Matt

On Tue, May 13, 2025 at 6:47 AM Andrew Lamb <al...@influxdata.com> wrote:
>
> Given the momentum I see in my day to day around the Parquet Variant type
> (Spark, Snowflake, parquet-java, iceberg, soon to be parquet-rs, etc) I am
> heavily in favor of making it a canonical extension type in Arrow that is
> zero-copy compatible with the Parquet Variant.
>
> Thank you for bringing up this important topic
>
> Andrew
>
> On Mon, May 12, 2025 at 2:21 PM Dewey Dunnington <de...@gmail.com>
> wrote:
>
> > > I think we should make clear that this extension type is for
> > > transporting Parquet Variants. If we were to design a Variant type
> > > specifically for Arrow, it would probably look a bit different
> >
> > That's a great point...there are definitely advantages to both: keeping the
> > spec identical to Parquet means it's easier to implement because it's
> > "just" applying a label to the default decode we get from Arrowish Parquet
> > readers and unlocks reading and writing without inventing new things.
> > Inventing an Arrow-native version that uses more Arrow concepts might be
> > faster for purely in-memory transformations, but if all we're ever doing
> > with it is encoding and decoding it from/into the Parquet version, is it
> > worth it? (Or can we defer the design of that type to a different time
> > based on the experience of the Parquet version?)
> >
> > On Mon, May 12, 2025 at 12:17 PM Felipe Oliveira Carvalho <
> > felipekde@gmail.com> wrote:
> >
> > > > As far as relying on union types, the reason we can't do so is because
> > > > the specific purpose of this Variant type is that we don't know the
> > > > types up front, it's dynamic.
> > >
> > > This is why "VARIANT" is a misnomer for this type. It's a DYNAMIC type,
> > not
> > > a VARIANT (a type that can be a sum of multiple types that are known).
> > >
> > > But the mis-use of VARIANT is already widespread, so we now end-up with
> > > UNIONS and VARIANTS on the same type system.
> > >
> > > --
> > > Felipe
> > >
> > > On Mon, May 12, 2025 at 1:21 PM Matt Topol <zo...@gmail.com>
> > wrote:
> > >
> > > > It's not just Parquet Variant, it's also Iceberg (which has
> > > > standardized on this) and Spark in-memory (where this encoding scheme
> > > > originated). It's actually an important aspect to this that it
> > > > supports easy and (mostly) zero-copy integration with spark's
> > > > in-memory representation of the Variant type.
> > > >
> > > > >  If we were to design a Variant type specifically for Arrow, it would
> > > > probably look a bit different (in particular, we would make a better
> > use
> > > of
> > > > validity bitmaps, and we would also rely on union types; besides, the
> > > > supported primitive types would reflect the Arrow type system).
> > > >
> > > > When coming up with this proposal I tried to think of ways I could
> > > > better leverage the validity bitmaps, and ultimately they are still
> > > > being used. The difficulty comes with having to represent the
> > > > difference between a field *missing* and a field being *present* but
> > > > null. If we could come up with a better way to represent that, I'd be
> > > > fine with using that in this proposal, but I couldn't think of a
> > > > better method than what they were using.
> > > >
> > > > As far as relying on union types, the reason we can't do so is because
> > > > the specific purpose of this Variant type is that we don't know the
> > > > types up front, it's dynamic. Also Arrow union types do not allow for
> > > > arbitrarily recursive structures, it's one of the issues that others
> > > > have run into with the Arrow union types. Another consideration is the
> > > > goal of projecting the shredded columns. Since we need to project
> > > > shredded columns, we'd only be able to use Sparse Unions, not dense
> > > > ones, and while it is well defined how to project the child of a
> > > > struct it's less defined how to project the child of a union.
> > > >
> > > > I'm still open to other possibilities in how we represent it, but
> > > > following the existing representation in this way seemed the best
> > > > route for interoperability.
> > > >
> > > > --Matt
> > > >
> > > > On Mon, May 12, 2025 at 2:47 AM Antoine Pitrou <an...@python.org>
> > > wrote:
> > > > >
> > > > >
> > > > > Hi Matt,
> > > > >
> > > > > Thanks for putting this together.
> > > > >
> > > > > I think we should make clear that this extension type is for
> > > > > transporting Parquet Variants. If we were to design a Variant type
> > > > > specifically for Arrow, it would probably look a bit different (in
> > > > > particular, we would make a better use of validity bitmaps, and we
> > > would
> > > > > also rely on union types; besides, the supported primitive types
> > would
> > > > > reflect the Arrow type system).
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > > >
> > > > >
> > > > > Le 09/05/2025 à 00:00, Matt Topol a écrit :
> > > > > > Hey All,
> > > > > >
> > > > > > There's been various discussions occurring on many different thread
> > > > > > locations (issues, PRs, and so on)[1][2][3], and more that I
> > haven't
> > > > > > linked to, concerning what a canonical Variant Extension Type for
> > > > > > Arrow might look like. As I've looked into implementing some
> > things,
> > > > > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > > > > communities as to what a good representation for Arrow Variant
> > would
> > > > > > be like in order to ensure good support and adoption.
> > > > > >
> > > > > > I also looked at the ClickHouse variant implementation [4]. The
> > > > > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union
> > > type,
> > > > > > so we don't need to do any extra work there to support it.
> > > > > >
> > > > > > So, after discussions and looking into the needs for engines and so
> > > > > > on, I've iterated and written up a proposal for what a Canonical
> > > > > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > > > > hoping that this can spark some discussion and comments on the
> > > > > > document. If there's relative consensus on it, then I'll work on
> > > > > > creating some implementations of it that I can use to formally
> > > propose
> > > > > > the addition to the Canonical Extensions.
> > > > > >
> > > > > > Please take a read and leave comments on the google doc or on this
> > > > > > thread. Thanks everyone!
> > > > > >
> > > > > > --Matt
> > > > > >
> > > > > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > > > > [2]: https://github.com/apache/arrow/issues/45937
> > > > > > [3]:
> > > > https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > > > > [4]:
> > > >
> > https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > > > > [5]:
> > > >
> > >
> > https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Andrew Lamb <al...@influxdata.com>.
Given the momentum I see in my day to day around the Parquet Variant type
(Spark, Snowflake, parquet-java, iceberg, soon to be parquet-rs, etc) I am
heavily in favor of making it a canonical extension type in Arrow that is
zero-copy compatible with the Parquet Variant.

Thank you for bringing up this important topic

Andrew

On Mon, May 12, 2025 at 2:21 PM Dewey Dunnington <de...@gmail.com>
wrote:

> > I think we should make clear that this extension type is for
> > transporting Parquet Variants. If we were to design a Variant type
> > specifically for Arrow, it would probably look a bit different
>
> That's a great point...there are definitely advantages to both: keeping the
> spec identical to Parquet means it's easier to implement because it's
> "just" applying a label to the default decode we get from Arrowish Parquet
> readers and unlocks reading and writing without inventing new things.
> Inventing an Arrow-native version that uses more Arrow concepts might be
> faster for purely in-memory transformations, but if all we're ever doing
> with it is encoding and decoding it from/into the Parquet version, is it
> worth it? (Or can we defer the design of that type to a different time
> based on the experience of the Parquet version?)
>
> On Mon, May 12, 2025 at 12:17 PM Felipe Oliveira Carvalho <
> felipekde@gmail.com> wrote:
>
> > > As far as relying on union types, the reason we can't do so is because
> > > the specific purpose of this Variant type is that we don't know the
> > > types up front, it's dynamic.
> >
> > This is why "VARIANT" is a misnomer for this type. It's a DYNAMIC type,
> not
> > a VARIANT (a type that can be a sum of multiple types that are known).
> >
> > But the mis-use of VARIANT is already widespread, so we now end-up with
> > UNIONS and VARIANTS on the same type system.
> >
> > --
> > Felipe
> >
> > On Mon, May 12, 2025 at 1:21 PM Matt Topol <zo...@gmail.com>
> wrote:
> >
> > > It's not just Parquet Variant, it's also Iceberg (which has
> > > standardized on this) and Spark in-memory (where this encoding scheme
> > > originated). It's actually an important aspect to this that it
> > > supports easy and (mostly) zero-copy integration with spark's
> > > in-memory representation of the Variant type.
> > >
> > > >  If we were to design a Variant type specifically for Arrow, it would
> > > probably look a bit different (in particular, we would make a better
> use
> > of
> > > validity bitmaps, and we would also rely on union types; besides, the
> > > supported primitive types would reflect the Arrow type system).
> > >
> > > When coming up with this proposal I tried to think of ways I could
> > > better leverage the validity bitmaps, and ultimately they are still
> > > being used. The difficulty comes with having to represent the
> > > difference between a field *missing* and a field being *present* but
> > > null. If we could come up with a better way to represent that, I'd be
> > > fine with using that in this proposal, but I couldn't think of a
> > > better method than what they were using.
> > >
> > > As far as relying on union types, the reason we can't do so is because
> > > the specific purpose of this Variant type is that we don't know the
> > > types up front, it's dynamic. Also Arrow union types do not allow for
> > > arbitrarily recursive structures, it's one of the issues that others
> > > have run into with the Arrow union types. Another consideration is the
> > > goal of projecting the shredded columns. Since we need to project
> > > shredded columns, we'd only be able to use Sparse Unions, not dense
> > > ones, and while it is well defined how to project the child of a
> > > struct it's less defined how to project the child of a union.
> > >
> > > I'm still open to other possibilities in how we represent it, but
> > > following the existing representation in this way seemed the best
> > > route for interoperability.
> > >
> > > --Matt
> > >
> > > On Mon, May 12, 2025 at 2:47 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > > >
> > > >
> > > > Hi Matt,
> > > >
> > > > Thanks for putting this together.
> > > >
> > > > I think we should make clear that this extension type is for
> > > > transporting Parquet Variants. If we were to design a Variant type
> > > > specifically for Arrow, it would probably look a bit different (in
> > > > particular, we would make a better use of validity bitmaps, and we
> > would
> > > > also rely on union types; besides, the supported primitive types
> would
> > > > reflect the Arrow type system).
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > > Le 09/05/2025 à 00:00, Matt Topol a écrit :
> > > > > Hey All,
> > > > >
> > > > > There's been various discussions occurring on many different thread
> > > > > locations (issues, PRs, and so on)[1][2][3], and more that I
> haven't
> > > > > linked to, concerning what a canonical Variant Extension Type for
> > > > > Arrow might look like. As I've looked into implementing some
> things,
> > > > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > > > communities as to what a good representation for Arrow Variant
> would
> > > > > be like in order to ensure good support and adoption.
> > > > >
> > > > > I also looked at the ClickHouse variant implementation [4]. The
> > > > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union
> > type,
> > > > > so we don't need to do any extra work there to support it.
> > > > >
> > > > > So, after discussions and looking into the needs for engines and so
> > > > > on, I've iterated and written up a proposal for what a Canonical
> > > > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > > > hoping that this can spark some discussion and comments on the
> > > > > document. If there's relative consensus on it, then I'll work on
> > > > > creating some implementations of it that I can use to formally
> > propose
> > > > > the addition to the Canonical Extensions.
> > > > >
> > > > > Please take a read and leave comments on the google doc or on this
> > > > > thread. Thanks everyone!
> > > > >
> > > > > --Matt
> > > > >
> > > > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > > > [2]: https://github.com/apache/arrow/issues/45937
> > > > > [3]:
> > > https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > > > [4]:
> > >
> https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > > > [5]:
> > >
> >
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> > > >
> > >
> >
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Dewey Dunnington <de...@gmail.com>.
> I think we should make clear that this extension type is for
> transporting Parquet Variants. If we were to design a Variant type
> specifically for Arrow, it would probably look a bit different

That's a great point...there are definitely advantages to both: keeping the
spec identical to Parquet means it's easier to implement because it's
"just" applying a label to the default decode we get from Arrowish Parquet
readers and unlocks reading and writing without inventing new things.
Inventing an Arrow-native version that uses more Arrow concepts might be
faster for purely in-memory transformations, but if all we're ever doing
with it is encoding and decoding it from/into the Parquet version, is it
worth it? (Or can we defer the design of that type to a different time
based on the experience of the Parquet version?)

On Mon, May 12, 2025 at 12:17 PM Felipe Oliveira Carvalho <
felipekde@gmail.com> wrote:

> > As far as relying on union types, the reason we can't do so is because
> > the specific purpose of this Variant type is that we don't know the
> > types up front, it's dynamic.
>
> This is why "VARIANT" is a misnomer for this type. It's a DYNAMIC type, not
> a VARIANT (a type that can be a sum of multiple types that are known).
>
> But the mis-use of VARIANT is already widespread, so we now end-up with
> UNIONS and VARIANTS on the same type system.
>
> --
> Felipe
>
> On Mon, May 12, 2025 at 1:21 PM Matt Topol <zo...@gmail.com> wrote:
>
> > It's not just Parquet Variant, it's also Iceberg (which has
> > standardized on this) and Spark in-memory (where this encoding scheme
> > originated). It's actually an important aspect to this that it
> > supports easy and (mostly) zero-copy integration with spark's
> > in-memory representation of the Variant type.
> >
> > >  If we were to design a Variant type specifically for Arrow, it would
> > probably look a bit different (in particular, we would make a better use
> of
> > validity bitmaps, and we would also rely on union types; besides, the
> > supported primitive types would reflect the Arrow type system).
> >
> > When coming up with this proposal I tried to think of ways I could
> > better leverage the validity bitmaps, and ultimately they are still
> > being used. The difficulty comes with having to represent the
> > difference between a field *missing* and a field being *present* but
> > null. If we could come up with a better way to represent that, I'd be
> > fine with using that in this proposal, but I couldn't think of a
> > better method than what they were using.
> >
> > As far as relying on union types, the reason we can't do so is because
> > the specific purpose of this Variant type is that we don't know the
> > types up front, it's dynamic. Also Arrow union types do not allow for
> > arbitrarily recursive structures, it's one of the issues that others
> > have run into with the Arrow union types. Another consideration is the
> > goal of projecting the shredded columns. Since we need to project
> > shredded columns, we'd only be able to use Sparse Unions, not dense
> > ones, and while it is well defined how to project the child of a
> > struct it's less defined how to project the child of a union.
> >
> > I'm still open to other possibilities in how we represent it, but
> > following the existing representation in this way seemed the best
> > route for interoperability.
> >
> > --Matt
> >
> > On Mon, May 12, 2025 at 2:47 AM Antoine Pitrou <an...@python.org>
> wrote:
> > >
> > >
> > > Hi Matt,
> > >
> > > Thanks for putting this together.
> > >
> > > I think we should make clear that this extension type is for
> > > transporting Parquet Variants. If we were to design a Variant type
> > > specifically for Arrow, it would probably look a bit different (in
> > > particular, we would make a better use of validity bitmaps, and we
> would
> > > also rely on union types; besides, the supported primitive types would
> > > reflect the Arrow type system).
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > > Le 09/05/2025 à 00:00, Matt Topol a écrit :
> > > > Hey All,
> > > >
> > > > There's been various discussions occurring on many different thread
> > > > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > > > linked to, concerning what a canonical Variant Extension Type for
> > > > Arrow might look like. As I've looked into implementing some things,
> > > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > > communities as to what a good representation for Arrow Variant would
> > > > be like in order to ensure good support and adoption.
> > > >
> > > > I also looked at the ClickHouse variant implementation [4]. The
> > > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union
> type,
> > > > so we don't need to do any extra work there to support it.
> > > >
> > > > So, after discussions and looking into the needs for engines and so
> > > > on, I've iterated and written up a proposal for what a Canonical
> > > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > > hoping that this can spark some discussion and comments on the
> > > > document. If there's relative consensus on it, then I'll work on
> > > > creating some implementations of it that I can use to formally
> propose
> > > > the addition to the Canonical Extensions.
> > > >
> > > > Please take a read and leave comments on the google doc or on this
> > > > thread. Thanks everyone!
> > > >
> > > > --Matt
> > > >
> > > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > > [2]: https://github.com/apache/arrow/issues/45937
> > > > [3]:
> > https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > > [4]:
> > https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > > [5]:
> >
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> > >
> >
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Felipe Oliveira Carvalho <fe...@gmail.com>.
> As far as relying on union types, the reason we can't do so is because
> the specific purpose of this Variant type is that we don't know the
> types up front, it's dynamic.

This is why "VARIANT" is a misnomer for this type. It's a DYNAMIC type, not
a VARIANT (a type that can be a sum of multiple types that are known).

But the mis-use of VARIANT is already widespread, so we now end-up with
UNIONS and VARIANTS on the same type system.

--
Felipe

On Mon, May 12, 2025 at 1:21 PM Matt Topol <zo...@gmail.com> wrote:

> It's not just Parquet Variant, it's also Iceberg (which has
> standardized on this) and Spark in-memory (where this encoding scheme
> originated). It's actually an important aspect to this that it
> supports easy and (mostly) zero-copy integration with spark's
> in-memory representation of the Variant type.
>
> >  If we were to design a Variant type specifically for Arrow, it would
> probably look a bit different (in particular, we would make a better use of
> validity bitmaps, and we would also rely on union types; besides, the
> supported primitive types would reflect the Arrow type system).
>
> When coming up with this proposal I tried to think of ways I could
> better leverage the validity bitmaps, and ultimately they are still
> being used. The difficulty comes with having to represent the
> difference between a field *missing* and a field being *present* but
> null. If we could come up with a better way to represent that, I'd be
> fine with using that in this proposal, but I couldn't think of a
> better method than what they were using.
>
> As far as relying on union types, the reason we can't do so is because
> the specific purpose of this Variant type is that we don't know the
> types up front, it's dynamic. Also Arrow union types do not allow for
> arbitrarily recursive structures, it's one of the issues that others
> have run into with the Arrow union types. Another consideration is the
> goal of projecting the shredded columns. Since we need to project
> shredded columns, we'd only be able to use Sparse Unions, not dense
> ones, and while it is well defined how to project the child of a
> struct it's less defined how to project the child of a union.
>
> I'm still open to other possibilities in how we represent it, but
> following the existing representation in this way seemed the best
> route for interoperability.
>
> --Matt
>
> On Mon, May 12, 2025 at 2:47 AM Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > Hi Matt,
> >
> > Thanks for putting this together.
> >
> > I think we should make clear that this extension type is for
> > transporting Parquet Variants. If we were to design a Variant type
> > specifically for Arrow, it would probably look a bit different (in
> > particular, we would make a better use of validity bitmaps, and we would
> > also rely on union types; besides, the supported primitive types would
> > reflect the Arrow type system).
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 09/05/2025 à 00:00, Matt Topol a écrit :
> > > Hey All,
> > >
> > > There's been various discussions occurring on many different thread
> > > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > > linked to, concerning what a canonical Variant Extension Type for
> > > Arrow might look like. As I've looked into implementing some things,
> > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > communities as to what a good representation for Arrow Variant would
> > > be like in order to ensure good support and adoption.
> > >
> > > I also looked at the ClickHouse variant implementation [4]. The
> > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> > > so we don't need to do any extra work there to support it.
> > >
> > > So, after discussions and looking into the needs for engines and so
> > > on, I've iterated and written up a proposal for what a Canonical
> > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > hoping that this can spark some discussion and comments on the
> > > document. If there's relative consensus on it, then I'll work on
> > > creating some implementations of it that I can use to formally propose
> > > the addition to the Canonical Extensions.
> > >
> > > Please take a read and leave comments on the google doc or on this
> > > thread. Thanks everyone!
> > >
> > > --Matt
> > >
> > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > [2]: https://github.com/apache/arrow/issues/45937
> > > [3]:
> https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > [4]:
> https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > [5]:
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> >
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Matt Topol <zo...@gmail.com>.
It's not just Parquet Variant, it's also Iceberg (which has
standardized on this) and Spark in-memory (where this encoding scheme
originated). It's actually an important aspect to this that it
supports easy and (mostly) zero-copy integration with spark's
in-memory representation of the Variant type.

>  If we were to design a Variant type specifically for Arrow, it would probably look a bit different (in particular, we would make a better use of validity bitmaps, and we would also rely on union types; besides, the supported primitive types would reflect the Arrow type system).

When coming up with this proposal I tried to think of ways I could
better leverage the validity bitmaps, and ultimately they are still
being used. The difficulty comes with having to represent the
difference between a field *missing* and a field being *present* but
null. If we could come up with a better way to represent that, I'd be
fine with using that in this proposal, but I couldn't think of a
better method than what they were using.

As far as relying on union types, the reason we can't do so is because
the specific purpose of this Variant type is that we don't know the
types up front, it's dynamic. Also Arrow union types do not allow for
arbitrarily recursive structures, it's one of the issues that others
have run into with the Arrow union types. Another consideration is the
goal of projecting the shredded columns. Since we need to project
shredded columns, we'd only be able to use Sparse Unions, not dense
ones, and while it is well defined how to project the child of a
struct it's less defined how to project the child of a union.

I'm still open to other possibilities in how we represent it, but
following the existing representation in this way seemed the best
route for interoperability.

--Matt

On Mon, May 12, 2025 at 2:47 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hi Matt,
>
> Thanks for putting this together.
>
> I think we should make clear that this extension type is for
> transporting Parquet Variants. If we were to design a Variant type
> specifically for Arrow, it would probably look a bit different (in
> particular, we would make a better use of validity bitmaps, and we would
> also rely on union types; besides, the supported primitive types would
> reflect the Arrow type system).
>
> Regards
>
> Antoine.
>
>
> Le 09/05/2025 à 00:00, Matt Topol a écrit :
> > Hey All,
> >
> > There's been various discussions occurring on many different thread
> > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > linked to, concerning what a canonical Variant Extension Type for
> > Arrow might look like. As I've looked into implementing some things,
> > I've also spoken with members of the Arrow, Iceberg and Parquet
> > communities as to what a good representation for Arrow Variant would
> > be like in order to ensure good support and adoption.
> >
> > I also looked at the ClickHouse variant implementation [4]. The
> > ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> > so we don't need to do any extra work there to support it.
> >
> > So, after discussions and looking into the needs for engines and so
> > on, I've iterated and written up a proposal for what a Canonical
> > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > hoping that this can spark some discussion and comments on the
> > document. If there's relative consensus on it, then I'll work on
> > creating some implementations of it that I can use to formally propose
> > the addition to the Canonical Extensions.
> >
> > Please take a read and leave comments on the google doc or on this
> > thread. Thanks everyone!
> >
> > --Matt
> >
> > [1]: https://github.com/apache/arrow-rs/issues/7063
> > [2]: https://github.com/apache/arrow/issues/45937
> > [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > [4]: https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > [5]: https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Antoine Pitrou <an...@python.org>.
Hi Matt,

Thanks for putting this together.

I think we should make clear that this extension type is for 
transporting Parquet Variants. If we were to design a Variant type 
specifically for Arrow, it would probably look a bit different (in 
particular, we would make a better use of validity bitmaps, and we would 
also rely on union types; besides, the supported primitive types would 
reflect the Arrow type system).

Regards

Antoine.


Le 09/05/2025 à 00:00, Matt Topol a écrit :
> Hey All,
> 
> There's been various discussions occurring on many different thread
> locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> linked to, concerning what a canonical Variant Extension Type for
> Arrow might look like. As I've looked into implementing some things,
> I've also spoken with members of the Arrow, Iceberg and Parquet
> communities as to what a good representation for Arrow Variant would
> be like in order to ensure good support and adoption.
> 
> I also looked at the ClickHouse variant implementation [4]. The
> ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> so we don't need to do any extra work there to support it.
> 
> So, after discussions and looking into the needs for engines and so
> on, I've iterated and written up a proposal for what a Canonical
> Variant Extension Type for Arrow could be in a google doc[5]. I'm
> hoping that this can spark some discussion and comments on the
> document. If there's relative consensus on it, then I'll work on
> creating some implementations of it that I can use to formally propose
> the addition to the Canonical Extensions.
> 
> Please take a read and leave comments on the google doc or on this
> thread. Thanks everyone!
> 
> --Matt
> 
> [1]: https://github.com/apache/arrow-rs/issues/7063
> [2]: https://github.com/apache/arrow/issues/45937
> [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> [4]: https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> [5]: https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing


Re: [DISCUSS] Arrow Variant Extension Type

Posted by Andrew Lamb <al...@influxdata.com>.
I am not sure a proper writeup is useful quite yet

I have filed a ticket to track the idea[1]. I'll report back here or on
that ticket as we work on implementing this in arrow-rs. Thank you

[1]: https://github.com/apache/arrow/issues/46908

On Wed, Jun 25, 2025 at 12:47 PM Matt Topol <zo...@gmail.com> wrote:

> We've merged the extension type into the Arrow Go implementation[1], I've
> been waiting for one of the other implementations to implement the proposal
> [2] before I go make a PR to add it to the docs in full. If you think it's
> worthwhile for me to start drafting up a PR to add to the Canonical
> Extensions right now then I'm happy to do so. I think most of the
> objections to using an extension type instead of a real type are answered
> or managed by the Proposal [2] and ensuring the extension type has
> appropriate functional support and methods.
>
> I think filing a ticket to track the work makes a lot of sense!
>
> --Matt
>
> [1]:
>
> https://github.com/apache/arrow-go/commit/5240503993cc0aa47554b932c341e4940ce42348
> [2]:
>
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
>
> On Wed, Jun 25, 2025 at 12:38 PM Andrew Lamb <al...@influxdata.com> wrote:
>
> > Did we ever decide that Variant will be a Arrow canonical extension type?
> >
> > I don't see it currently listed in the docs [1] however an extension type
> > maybe was added to the C++ implementation in [2] (sorry I am not
> > familiar with that codebase to be sure)
> >
> > As I think was mentioned elsewhere there is also a github discussion
> about
> > adding Variant as a real type[3] that may also be relevant, from Curt.
> >
> > If this is the direction we are heading I will be happy to file a ticket
> to
> > track the work
> >
> > Andrew
> >
> > [1]:
> >
> >
> https://arrow.apache.org/docs/format/CanonicalExtensions.html#canonical-extension-types
> > [2]: https://github.com/apache/arrow/pull/45375/files
> > [3]: https://github.com/apache/arrow/issues/42069
> >
> > On Wed, May 21, 2025 at 4:43 AM wish maple <ma...@gmail.com>
> wrote:
> >
> > > When I went through the parquet variant spec, I found that an arrow
> > > extension type might be a must because decoding the parquet row
> > > by row is so inefficient.
> > >
> > > I've draft a decoding tool in parquet c++ and ready for review now [1]
> > >
> > > [1] https://github.com/apache/arrow/pull/46372
> > >
> > > Best,
> > > Xuwei Fu
> > >
> > > Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:
> > >
> > > > Hey All,
> > > >
> > > > There's been various discussions occurring on many different thread
> > > > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > > > linked to, concerning what a canonical Variant Extension Type for
> > > > Arrow might look like. As I've looked into implementing some things,
> > > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > > communities as to what a good representation for Arrow Variant would
> > > > be like in order to ensure good support and adoption.
> > > >
> > > > I also looked at the ClickHouse variant implementation [4]. The
> > > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union
> type,
> > > > so we don't need to do any extra work there to support it.
> > > >
> > > > So, after discussions and looking into the needs for engines and so
> > > > on, I've iterated and written up a proposal for what a Canonical
> > > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > > hoping that this can spark some discussion and comments on the
> > > > document. If there's relative consensus on it, then I'll work on
> > > > creating some implementations of it that I can use to formally
> propose
> > > > the addition to the Canonical Extensions.
> > > >
> > > > Please take a read and leave comments on the google doc or on this
> > > > thread. Thanks everyone!
> > > >
> > > > --Matt
> > > >
> > > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > > [2]: https://github.com/apache/arrow/issues/45937
> > > > [3]:
> > https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > > [4]:
> > > >
> > https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > > [5]:
> > > >
> > >
> >
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> > > >
> > >
> >
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Matt Topol <zo...@gmail.com>.
We've merged the extension type into the Arrow Go implementation[1], I've
been waiting for one of the other implementations to implement the proposal
[2] before I go make a PR to add it to the docs in full. If you think it's
worthwhile for me to start drafting up a PR to add to the Canonical
Extensions right now then I'm happy to do so. I think most of the
objections to using an extension type instead of a real type are answered
or managed by the Proposal [2] and ensuring the extension type has
appropriate functional support and methods.

I think filing a ticket to track the work makes a lot of sense!

--Matt

[1]:
https://github.com/apache/arrow-go/commit/5240503993cc0aa47554b932c341e4940ce42348
[2]:
https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing

On Wed, Jun 25, 2025 at 12:38 PM Andrew Lamb <al...@influxdata.com> wrote:

> Did we ever decide that Variant will be a Arrow canonical extension type?
>
> I don't see it currently listed in the docs [1] however an extension type
> maybe was added to the C++ implementation in [2] (sorry I am not
> familiar with that codebase to be sure)
>
> As I think was mentioned elsewhere there is also a github discussion about
> adding Variant as a real type[3] that may also be relevant, from Curt.
>
> If this is the direction we are heading I will be happy to file a ticket to
> track the work
>
> Andrew
>
> [1]:
>
> https://arrow.apache.org/docs/format/CanonicalExtensions.html#canonical-extension-types
> [2]: https://github.com/apache/arrow/pull/45375/files
> [3]: https://github.com/apache/arrow/issues/42069
>
> On Wed, May 21, 2025 at 4:43 AM wish maple <ma...@gmail.com> wrote:
>
> > When I went through the parquet variant spec, I found that an arrow
> > extension type might be a must because decoding the parquet row
> > by row is so inefficient.
> >
> > I've draft a decoding tool in parquet c++ and ready for review now [1]
> >
> > [1] https://github.com/apache/arrow/pull/46372
> >
> > Best,
> > Xuwei Fu
> >
> > Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:
> >
> > > Hey All,
> > >
> > > There's been various discussions occurring on many different thread
> > > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > > linked to, concerning what a canonical Variant Extension Type for
> > > Arrow might look like. As I've looked into implementing some things,
> > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > communities as to what a good representation for Arrow Variant would
> > > be like in order to ensure good support and adoption.
> > >
> > > I also looked at the ClickHouse variant implementation [4]. The
> > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> > > so we don't need to do any extra work there to support it.
> > >
> > > So, after discussions and looking into the needs for engines and so
> > > on, I've iterated and written up a proposal for what a Canonical
> > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > hoping that this can spark some discussion and comments on the
> > > document. If there's relative consensus on it, then I'll work on
> > > creating some implementations of it that I can use to formally propose
> > > the addition to the Canonical Extensions.
> > >
> > > Please take a read and leave comments on the google doc or on this
> > > thread. Thanks everyone!
> > >
> > > --Matt
> > >
> > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > [2]: https://github.com/apache/arrow/issues/45937
> > > [3]:
> https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > [4]:
> > >
> https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > [5]:
> > >
> >
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> > >
> >
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Antoine Pitrou <an...@python.org>.
No, I think it's fine. Now someone has to write an actual PR for the 
format docs, I guess.

Thanks

Antoine.


Le 01/07/2025 à 17:44, Andrew Lamb a écrit :
>> The problem is that the discussion is still framed as "Arrow Variant"
>> type (see mail subject line) but most people seem to be thinking of
>> canonicalizing a Parquet Variant extension type in Arrow.
> 
> I have renamed the ticket[1] to "[Format] Add an Arrow Canonical Extension
> Type for Parquet Variant #46908" to try and reduce the confusion
> 
> Antoine, do you have any other recommendations on how to avoid confusion
> other than being more precise with the naming?
> 
> Andrew
> 
> [1]: https://github.com/apache/arrow/issues/46908
> 
> 
> On Thu, Jun 26, 2025 at 8:43 AM Joris Van den Bossche <
> jorisvandenbossche@gmail.com> wrote:
> 
>> Note that the extension type that was merged in Go
>> (
>> https://github.com/apache/arrow-go/blob/c542dd68e2757122ce8ffc15936f2df46664c30c/arrow/extensions/variant.go#L170
>> )
>> and also the one used in Parquet C++ in the arrow repo is using the
>> name "parquet.variant", not "arrow.variant".
>>
>> That could help frame it as "Parquet variant" compatible instead of
>> *the* Arrow variant type. But from the discussion here (or the google
>> doc), it was not clear to me that this is the name being used in the
>> current implementations, and the proposal is to follow those
>> implementations or change them to "arrow.variant" once that would be
>> voted upon.
>>
>> Joris
>>
>> On Thu, 26 Jun 2025 at 13:58, Antoine Pitrou <an...@python.org> wrote:
>>>
>>>
>>> The problem is that the discussion is still framed as "Arrow Variant"
>>> type (see mail subject line) but most people seem to be thinking of
>>> canonicalizing a Parquet Variant extension type in Arrow.
>>>
>>> That confusion should be cleared before we think of moving any further.
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> On Wed, 25 Jun 2025 12:38:21 -0400
>>> Andrew Lamb <al...@influxdata.com> wrote:
>>>> Did we ever decide that Variant will be a Arrow canonical extension
>> type?
>>>>
>>>> I don't see it currently listed in the docs [1] however an extension
>> type
>>>> maybe was added to the C++ implementation in [2] (sorry I am not
>>>> familiar with that codebase to be sure)
>>>>
>>>> As I think was mentioned elsewhere there is also a github discussion
>> about
>>>> adding Variant as a real type[3] that may also be relevant, from Curt.
>>>>
>>>> If this is the direction we are heading I will be happy to file a
>> ticket to
>>>> track the work
>>>>
>>>> Andrew
>>>>
>>>> [1]:
>>>>
>> https://arrow.apache.org/docs/format/CanonicalExtensions.html#canonical-extension-types
>>>> [2]: https://github.com/apache/arrow/pull/45375/files
>>>> [3]: https://github.com/apache/arrow/issues/42069
>>>>
>>>> On Wed, May 21, 2025 at 4:43 AM wish maple <ma...@gmail.com>
>> wrote:
>>>>
>>>>> When I went through the parquet variant spec, I found that an arrow
>>>>> extension type might be a must because decoding the parquet row
>>>>> by row is so inefficient.
>>>>>
>>>>> I've draft a decoding tool in parquet c++ and ready for review now
>> [1]
>>>>>
>>>>> [1] https://github.com/apache/arrow/pull/46372
>>>>>
>>>>> Best,
>>>>> Xuwei Fu
>>>>>
>>>>> Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:
>>>>>
>>>>>> Hey All,
>>>>>>
>>>>>> There's been various discussions occurring on many different thread
>>>>>> locations (issues, PRs, and so on)[1][2][3], and more that I
>> haven't
>>>>>> linked to, concerning what a canonical Variant Extension Type for
>>>>>> Arrow might look like. As I've looked into implementing some
>> things,
>>>>>> I've also spoken with members of the Arrow, Iceberg and Parquet
>>>>>> communities as to what a good representation for Arrow Variant
>> would
>>>>>> be like in order to ensure good support and adoption.
>>>>>>
>>>>>> I also looked at the ClickHouse variant implementation [4]. The
>>>>>> ClickHouse Variant is nearly equivalent to the Arrow Dense Union
>> type,
>>>>>> so we don't need to do any extra work there to support it.
>>>>>>
>>>>>> So, after discussions and looking into the needs for engines and so
>>>>>> on, I've iterated and written up a proposal for what a Canonical
>>>>>> Variant Extension Type for Arrow could be in a google doc[5]. I'm
>>>>>> hoping that this can spark some discussion and comments on the
>>>>>> document. If there's relative consensus on it, then I'll work on
>>>>>> creating some implementations of it that I can use to formally
>> propose
>>>>>> the addition to the Canonical Extensions.
>>>>>>
>>>>>> Please take a read and leave comments on the google doc or on this
>>>>>> thread. Thanks everyone!
>>>>>>
>>>>>> --Matt
>>>>>>
>>>>>> [1]: https://github.com/apache/arrow-rs/issues/7063
>>>>>> [2]: https://github.com/apache/arrow/issues/45937
>>>>>> [3]:
>> https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
>>>>>> [4]:
>>>>>>
>> https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
>>>>>> [5]:
>>>>>>
>>>>>
>> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
> 


Re: [DISCUSS] Arrow Variant Extension Type

Posted by Andrew Lamb <al...@influxdata.com>.
> The problem is that the discussion is still framed as "Arrow Variant"
> type (see mail subject line) but most people seem to be thinking of
> canonicalizing a Parquet Variant extension type in Arrow.

I have renamed the ticket[1] to "[Format] Add an Arrow Canonical Extension
Type for Parquet Variant #46908" to try and reduce the confusion

Antoine, do you have any other recommendations on how to avoid confusion
other than being more precise with the naming?

Andrew

[1]: https://github.com/apache/arrow/issues/46908


On Thu, Jun 26, 2025 at 8:43 AM Joris Van den Bossche <
jorisvandenbossche@gmail.com> wrote:

> Note that the extension type that was merged in Go
> (
> https://github.com/apache/arrow-go/blob/c542dd68e2757122ce8ffc15936f2df46664c30c/arrow/extensions/variant.go#L170
> )
> and also the one used in Parquet C++ in the arrow repo is using the
> name "parquet.variant", not "arrow.variant".
>
> That could help frame it as "Parquet variant" compatible instead of
> *the* Arrow variant type. But from the discussion here (or the google
> doc), it was not clear to me that this is the name being used in the
> current implementations, and the proposal is to follow those
> implementations or change them to "arrow.variant" once that would be
> voted upon.
>
> Joris
>
> On Thu, 26 Jun 2025 at 13:58, Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > The problem is that the discussion is still framed as "Arrow Variant"
> > type (see mail subject line) but most people seem to be thinking of
> > canonicalizing a Parquet Variant extension type in Arrow.
> >
> > That confusion should be cleared before we think of moving any further.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > On Wed, 25 Jun 2025 12:38:21 -0400
> > Andrew Lamb <al...@influxdata.com> wrote:
> > > Did we ever decide that Variant will be a Arrow canonical extension
> type?
> > >
> > > I don't see it currently listed in the docs [1] however an extension
> type
> > > maybe was added to the C++ implementation in [2] (sorry I am not
> > > familiar with that codebase to be sure)
> > >
> > > As I think was mentioned elsewhere there is also a github discussion
> about
> > > adding Variant as a real type[3] that may also be relevant, from Curt.
> > >
> > > If this is the direction we are heading I will be happy to file a
> ticket to
> > > track the work
> > >
> > > Andrew
> > >
> > > [1]:
> > >
> https://arrow.apache.org/docs/format/CanonicalExtensions.html#canonical-extension-types
> > > [2]: https://github.com/apache/arrow/pull/45375/files
> > > [3]: https://github.com/apache/arrow/issues/42069
> > >
> > > On Wed, May 21, 2025 at 4:43 AM wish maple <ma...@gmail.com>
> wrote:
> > >
> > > > When I went through the parquet variant spec, I found that an arrow
> > > > extension type might be a must because decoding the parquet row
> > > > by row is so inefficient.
> > > >
> > > > I've draft a decoding tool in parquet c++ and ready for review now
> [1]
> > > >
> > > > [1] https://github.com/apache/arrow/pull/46372
> > > >
> > > > Best,
> > > > Xuwei Fu
> > > >
> > > > Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:
> > > >
> > > > > Hey All,
> > > > >
> > > > > There's been various discussions occurring on many different thread
> > > > > locations (issues, PRs, and so on)[1][2][3], and more that I
> haven't
> > > > > linked to, concerning what a canonical Variant Extension Type for
> > > > > Arrow might look like. As I've looked into implementing some
> things,
> > > > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > > > communities as to what a good representation for Arrow Variant
> would
> > > > > be like in order to ensure good support and adoption.
> > > > >
> > > > > I also looked at the ClickHouse variant implementation [4]. The
> > > > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union
> type,
> > > > > so we don't need to do any extra work there to support it.
> > > > >
> > > > > So, after discussions and looking into the needs for engines and so
> > > > > on, I've iterated and written up a proposal for what a Canonical
> > > > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > > > hoping that this can spark some discussion and comments on the
> > > > > document. If there's relative consensus on it, then I'll work on
> > > > > creating some implementations of it that I can use to formally
> propose
> > > > > the addition to the Canonical Extensions.
> > > > >
> > > > > Please take a read and leave comments on the google doc or on this
> > > > > thread. Thanks everyone!
> > > > >
> > > > > --Matt
> > > > >
> > > > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > > > [2]: https://github.com/apache/arrow/issues/45937
> > > > > [3]:
> https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > > > [4]:
> > > > >
> https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > > > [5]:
> > > > >
> > > >
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> > > > >
> > > >
> > >
> >
> >
> >
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Joris Van den Bossche <jo...@gmail.com>.
Note that the extension type that was merged in Go
(https://github.com/apache/arrow-go/blob/c542dd68e2757122ce8ffc15936f2df46664c30c/arrow/extensions/variant.go#L170)
and also the one used in Parquet C++ in the arrow repo is using the
name "parquet.variant", not "arrow.variant".

That could help frame it as "Parquet variant" compatible instead of
*the* Arrow variant type. But from the discussion here (or the google
doc), it was not clear to me that this is the name being used in the
current implementations, and the proposal is to follow those
implementations or change them to "arrow.variant" once that would be
voted upon.

Joris

On Thu, 26 Jun 2025 at 13:58, Antoine Pitrou <an...@python.org> wrote:
>
>
> The problem is that the discussion is still framed as "Arrow Variant"
> type (see mail subject line) but most people seem to be thinking of
> canonicalizing a Parquet Variant extension type in Arrow.
>
> That confusion should be cleared before we think of moving any further.
>
> Regards
>
> Antoine.
>
>
> On Wed, 25 Jun 2025 12:38:21 -0400
> Andrew Lamb <al...@influxdata.com> wrote:
> > Did we ever decide that Variant will be a Arrow canonical extension type?
> >
> > I don't see it currently listed in the docs [1] however an extension type
> > maybe was added to the C++ implementation in [2] (sorry I am not
> > familiar with that codebase to be sure)
> >
> > As I think was mentioned elsewhere there is also a github discussion about
> > adding Variant as a real type[3] that may also be relevant, from Curt.
> >
> > If this is the direction we are heading I will be happy to file a ticket to
> > track the work
> >
> > Andrew
> >
> > [1]:
> > https://arrow.apache.org/docs/format/CanonicalExtensions.html#canonical-extension-types
> > [2]: https://github.com/apache/arrow/pull/45375/files
> > [3]: https://github.com/apache/arrow/issues/42069
> >
> > On Wed, May 21, 2025 at 4:43 AM wish maple <ma...@gmail.com> wrote:
> >
> > > When I went through the parquet variant spec, I found that an arrow
> > > extension type might be a must because decoding the parquet row
> > > by row is so inefficient.
> > >
> > > I've draft a decoding tool in parquet c++ and ready for review now [1]
> > >
> > > [1] https://github.com/apache/arrow/pull/46372
> > >
> > > Best,
> > > Xuwei Fu
> > >
> > > Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:
> > >
> > > > Hey All,
> > > >
> > > > There's been various discussions occurring on many different thread
> > > > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > > > linked to, concerning what a canonical Variant Extension Type for
> > > > Arrow might look like. As I've looked into implementing some things,
> > > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > > communities as to what a good representation for Arrow Variant would
> > > > be like in order to ensure good support and adoption.
> > > >
> > > > I also looked at the ClickHouse variant implementation [4]. The
> > > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> > > > so we don't need to do any extra work there to support it.
> > > >
> > > > So, after discussions and looking into the needs for engines and so
> > > > on, I've iterated and written up a proposal for what a Canonical
> > > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > > hoping that this can spark some discussion and comments on the
> > > > document. If there's relative consensus on it, then I'll work on
> > > > creating some implementations of it that I can use to formally propose
> > > > the addition to the Canonical Extensions.
> > > >
> > > > Please take a read and leave comments on the google doc or on this
> > > > thread. Thanks everyone!
> > > >
> > > > --Matt
> > > >
> > > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > > [2]: https://github.com/apache/arrow/issues/45937
> > > > [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > > [4]:
> > > > https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > > [5]:
> > > >
> > > https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> > > >
> > >
> >
>
>
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by Antoine Pitrou <an...@python.org>.
The problem is that the discussion is still framed as "Arrow Variant"
type (see mail subject line) but most people seem to be thinking of
canonicalizing a Parquet Variant extension type in Arrow.

That confusion should be cleared before we think of moving any further.

Regards

Antoine.


On Wed, 25 Jun 2025 12:38:21 -0400
Andrew Lamb <al...@influxdata.com> wrote:
> Did we ever decide that Variant will be a Arrow canonical extension type?
> 
> I don't see it currently listed in the docs [1] however an extension type
> maybe was added to the C++ implementation in [2] (sorry I am not
> familiar with that codebase to be sure)
> 
> As I think was mentioned elsewhere there is also a github discussion about
> adding Variant as a real type[3] that may also be relevant, from Curt.
> 
> If this is the direction we are heading I will be happy to file a ticket to
> track the work
> 
> Andrew
> 
> [1]:
> https://arrow.apache.org/docs/format/CanonicalExtensions.html#canonical-extension-types
> [2]: https://github.com/apache/arrow/pull/45375/files
> [3]: https://github.com/apache/arrow/issues/42069
> 
> On Wed, May 21, 2025 at 4:43 AM wish maple <ma...@gmail.com> wrote:
> 
> > When I went through the parquet variant spec, I found that an arrow
> > extension type might be a must because decoding the parquet row
> > by row is so inefficient.
> >
> > I've draft a decoding tool in parquet c++ and ready for review now [1]
> >
> > [1] https://github.com/apache/arrow/pull/46372
> >
> > Best,
> > Xuwei Fu
> >
> > Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:
> >  
> > > Hey All,
> > >
> > > There's been various discussions occurring on many different thread
> > > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > > linked to, concerning what a canonical Variant Extension Type for
> > > Arrow might look like. As I've looked into implementing some things,
> > > I've also spoken with members of the Arrow, Iceberg and Parquet
> > > communities as to what a good representation for Arrow Variant would
> > > be like in order to ensure good support and adoption.
> > >
> > > I also looked at the ClickHouse variant implementation [4]. The
> > > ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> > > so we don't need to do any extra work there to support it.
> > >
> > > So, after discussions and looking into the needs for engines and so
> > > on, I've iterated and written up a proposal for what a Canonical
> > > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > > hoping that this can spark some discussion and comments on the
> > > document. If there's relative consensus on it, then I'll work on
> > > creating some implementations of it that I can use to formally propose
> > > the addition to the Canonical Extensions.
> > >
> > > Please take a read and leave comments on the google doc or on this
> > > thread. Thanks everyone!
> > >
> > > --Matt
> > >
> > > [1]: https://github.com/apache/arrow-rs/issues/7063
> > > [2]: https://github.com/apache/arrow/issues/45937
> > > [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > > [4]:
> > > https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > > [5]:
> > >  
> > https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing  
> > >  
> >  
> 




Re: [DISCUSS] Arrow Variant Extension Type

Posted by Andrew Lamb <al...@influxdata.com>.
Did we ever decide that Variant will be a Arrow canonical extension type?

I don't see it currently listed in the docs [1] however an extension type
maybe was added to the C++ implementation in [2] (sorry I am not
familiar with that codebase to be sure)

As I think was mentioned elsewhere there is also a github discussion about
adding Variant as a real type[3] that may also be relevant, from Curt.

If this is the direction we are heading I will be happy to file a ticket to
track the work

Andrew

[1]:
https://arrow.apache.org/docs/format/CanonicalExtensions.html#canonical-extension-types
[2]: https://github.com/apache/arrow/pull/45375/files
[3]: https://github.com/apache/arrow/issues/42069

On Wed, May 21, 2025 at 4:43 AM wish maple <ma...@gmail.com> wrote:

> When I went through the parquet variant spec, I found that an arrow
> extension type might be a must because decoding the parquet row
> by row is so inefficient.
>
> I've draft a decoding tool in parquet c++ and ready for review now [1]
>
> [1] https://github.com/apache/arrow/pull/46372
>
> Best,
> Xuwei Fu
>
> Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:
>
> > Hey All,
> >
> > There's been various discussions occurring on many different thread
> > locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> > linked to, concerning what a canonical Variant Extension Type for
> > Arrow might look like. As I've looked into implementing some things,
> > I've also spoken with members of the Arrow, Iceberg and Parquet
> > communities as to what a good representation for Arrow Variant would
> > be like in order to ensure good support and adoption.
> >
> > I also looked at the ClickHouse variant implementation [4]. The
> > ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> > so we don't need to do any extra work there to support it.
> >
> > So, after discussions and looking into the needs for engines and so
> > on, I've iterated and written up a proposal for what a Canonical
> > Variant Extension Type for Arrow could be in a google doc[5]. I'm
> > hoping that this can spark some discussion and comments on the
> > document. If there's relative consensus on it, then I'll work on
> > creating some implementations of it that I can use to formally propose
> > the addition to the Canonical Extensions.
> >
> > Please take a read and leave comments on the google doc or on this
> > thread. Thanks everyone!
> >
> > --Matt
> >
> > [1]: https://github.com/apache/arrow-rs/issues/7063
> > [2]: https://github.com/apache/arrow/issues/45937
> > [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> > [4]:
> > https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> > [5]:
> >
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
> >
>

Re: [DISCUSS] Arrow Variant Extension Type

Posted by wish maple <ma...@gmail.com>.
When I went through the parquet variant spec, I found that an arrow
extension type might be a must because decoding the parquet row
by row is so inefficient.

I've draft a decoding tool in parquet c++ and ready for review now [1]

[1] https://github.com/apache/arrow/pull/46372

Best,
Xuwei Fu

Matt Topol <zo...@gmail.com> 于2025年5月9日周五 06:03写道:

> Hey All,
>
> There's been various discussions occurring on many different thread
> locations (issues, PRs, and so on)[1][2][3], and more that I haven't
> linked to, concerning what a canonical Variant Extension Type for
> Arrow might look like. As I've looked into implementing some things,
> I've also spoken with members of the Arrow, Iceberg and Parquet
> communities as to what a good representation for Arrow Variant would
> be like in order to ensure good support and adoption.
>
> I also looked at the ClickHouse variant implementation [4]. The
> ClickHouse Variant is nearly equivalent to the Arrow Dense Union type,
> so we don't need to do any extra work there to support it.
>
> So, after discussions and looking into the needs for engines and so
> on, I've iterated and written up a proposal for what a Canonical
> Variant Extension Type for Arrow could be in a google doc[5]. I'm
> hoping that this can spark some discussion and comments on the
> document. If there's relative consensus on it, then I'll work on
> creating some implementations of it that I can use to formally propose
> the addition to the Canonical Extensions.
>
> Please take a read and leave comments on the google doc or on this
> thread. Thanks everyone!
>
> --Matt
>
> [1]: https://github.com/apache/arrow-rs/issues/7063
> [2]: https://github.com/apache/arrow/issues/45937
> [3]: https://github.com/apache/arrow/pull/45375#issuecomment-2649807352
> [4]:
> https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse
> [5]:
> https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
>