You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ramin Gharib <ra...@gmail.com> on 2026/08/04 10:28:49 UTC
[DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Hi everyone,
I'd like to start a discussion on FLIP-604: Complete VARIANT
Primitive Coverage with UUID and Timestamps. [1]
FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements Parquet
Variant primitive codes 0 through 16. The open Variant spec goes
through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
written by Spark, Iceberg, or any Parquet writer that holds one of
these values cannot be read by Flink today, which breaks the
lakehouse round-trip VARIANT was meant to support.
This FLIP completes the primitive set. TIME and the nanosecond
timestamps reuse types Flink already has. UUID needs a type to land
on, so the FLIP also adds a native 16-byte UUID type as a first-class
SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
UUID_V7 generator functions.
One point worth noting: no Calcite upgrade is required. Calcite has
had a UUID type since 1.39, the same release FLIP-521 adopted for
VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
matches every engine with a native UUID type, including PostgreSQL,
Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
Looking forward to your feedback.
Best,
Ramin Gharib
[1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
[2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Posted by Ramin Gharib <ra...@gmail.com>.
Hi Mate,
Agreed on all points. Failing made the cast behavior of the VARIANT odd, so
I've updated the FLIP to truncate. CAST(variant AS TIME) now truncates the
sub-millisecond digits, matching TIMESTAMP(9) → TIMESTAMP(3) and
CAST(string AS TIME). Same fot the case of casting a nanosecond Variant
timestamp to a lower precision, e.g. CAST(variant AS TIMESTAMP(3)),
truncates rather than fails. The full precision stays in the VARIANT bytes
and is reachable through the flink-core Variant API.
This truncation behavior is already implemented in my variant-to-primitive
cast PR [1].
Thanks for the careful review.
Best,
Ramin
[1] https://github.com/apache/flink/pull/28928
On Wed, Aug 5, 2026 at 2:53 PM Mate Czagany <cz...@gmail.com> wrote:
> Hi Ramin,
>
> Thank you for working on this. Looks good overall, one question on the cast
> semantics for VARIANT TIME.
>
> The FLIP says CAST(varant AS TIME) fails when the value has sub-millisecond
> digits. I think that would make VARIANT the only cast where narrowing the
> precision fails instead of truncating. Everywhere else Flink silently
> truncates: TIMESTAMP(9) -> TIMESTAMP(3) truncates [1],
> CAST('12:34:56.123456789' AS TIME) drops the fraction [2]. In CastRulesTest
> there is also a reference to an improvement ticket to support TIME type
> precision beyond milliseconds [3].
>
> This leads to odd cases: CAST(v AS TIME) would fail while CAST(CAST(v AS
> STRING) AS TIME) succeeds on the same value. And the sibling case is
> undefined in the FLIP: what does CAST(variant AS TIMESTAMP(3)) do when the
> variant holds a nanosecond timestamp. Truncate like the existing rule, or
> faillike the new TIME rule?
>
> I'd suggest truncating for consistency, and treating FLINK-39214 [3] as the
> proper long-term fix that removes the TIME gap entirely.
>
> Best,
> Mate
>
> [1]
>
> https://github.com/apache/flink/blob/33a198ae41ca1f11a8a8f313a5295551d1d591df/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/TimestampToTimestampCastRule.java#L95
> [2]
>
> https://github.com/apache/flink/blob/33a198ae41ca1f11a8a8f313a5295551d1d591df/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java#L500
> [3] https://issues.apache.org/jira/browse/FLINK-39214
>
>
> On Tue, Aug 4, 2026 at 4:22 PM Timo Walther <tw...@apache.org> wrote:
>
> > Thanks for the explanation Ramin. LGTM
> >
> > Cheers,
> > Timo
> >
> > On 04.08.26 15:27, Ramin Gharib wrote:
> > > Hello Timo,
> > > Thanks for looking into this and the feedback.
> > >
> > > Yes, for SQL: a Variant TIME lands as TIME(3), and per the FLIP a
> > sub-millisecond value fails the cast instead of truncating.
> > > Small correction on the DataStream side: the ceiling for TIME is
> micros,
> > not nanos. Variant code 17 is TIME(isAdjustedToUTC=false, MICROS) and the
> > spec has no nanosecond TIME, so getTime() surfaces at most precision 6.
> > >
> > > Nanosecond precision (9) applies to codes 18 and 19, the nanosecond
> > timestamps, not to TIME. TIMESTAMP(9) and TIMESTAMP_LTZ(9) already cover
> > them, so both SQL and DataStream keep the full nanoseconds. TIME is the
> > only code with a gap, since its Variant precision is micros (6) while SQL
> > TIME caps at millis (3).
> > >
> > > I've updated the FLIP to make this explicit (the TIME(3) mapping in the
> > table plus a note on the SQL-vs-flink-core precision).
> > >
> > > Best,
> > >
> > > Ramin
> > >
> > > From: Timo Walther <tw...@apache.org>
> > > Date: Tuesday, 4. August 2026 at 12:59
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage
> > with UUID and Timestamps
> > >
> > > Hi Ramin,
> > >
> > > thank you for evolving the VARIANT type further. I fully support the
> > FLIP.
> > >
> > > The ask for a UUID data type came up a couple of times in the past
> > > because UUIDs are in almost every data pipeline.
> > >
> > > Also full timestamp and time support incl nano second precision is a
> > > topic that always pops up and support for it avoids friction.
> > >
> > > One question the statement: "time without time zone (us)"
> > >
> > > Since Flink SQL only supports TIME(3), I guess we limit VARIANT also to
> > > millis within the SQL runtime, right? For DataStream API we can support
> > > 9, right?
> > >
> > > Cheers,
> > > Timo
> > >
> > > On 04.08.26 12:28, Ramin Gharib wrote:
> > >> Hi everyone,
> > >>
> > >> I'd like to start a discussion on FLIP-604: Complete VARIANT
> > >> Primitive Coverage with UUID and Timestamps. [1]
> > >>
> > >> FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements
> > Parquet
> > >> Variant primitive codes 0 through 16. The open Variant spec goes
> > >> through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
> > >> written by Spark, Iceberg, or any Parquet writer that holds one of
> > >> these values cannot be read by Flink today, which breaks the
> > >> lakehouse round-trip VARIANT was meant to support.
> > >>
> > >> This FLIP completes the primitive set. TIME and the nanosecond
> > >> timestamps reuse types Flink already has. UUID needs a type to land
> > >> on, so the FLIP also adds a native 16-byte UUID type as a first-class
> > >> SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
> > >> UUID_V7 generator functions.
> > >>
> > >> One point worth noting: no Calcite upgrade is required. Calcite has
> > >> had a UUID type since 1.39, the same release FLIP-521 adopted for
> > >> VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
> > >> matches every engine with a native UUID type, including PostgreSQL,
> > >> Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
> > >>
> > >> Looking forward to your feedback.
> > >>
> > >> Best,
> > >> Ramin Gharib
> > >>
> > >> [1]
> >
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
> > >> [2]
> >
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
> > >>
> > >>
> > >
> > >
> >
> >
>
Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Posted by Gustavo de Morais <gu...@gmail.com>.
Hey Ramin,
Thanks for working on making the VARIANT type support complete. The FLIP
looks good and is a core missing part to make the primitive types support
complete. +1
Kind regards,
Gustavo
On Wed, 5 Aug 2026 at 14:53, Mate Czagany <cz...@gmail.com> wrote:
> Hi Ramin,
>
> Thank you for working on this. Looks good overall, one question on the cast
> semantics for VARIANT TIME.
>
> The FLIP says CAST(varant AS TIME) fails when the value has sub-millisecond
> digits. I think that would make VARIANT the only cast where narrowing the
> precision fails instead of truncating. Everywhere else Flink silently
> truncates: TIMESTAMP(9) -> TIMESTAMP(3) truncates [1],
> CAST('12:34:56.123456789' AS TIME) drops the fraction [2]. In CastRulesTest
> there is also a reference to an improvement ticket to support TIME type
> precision beyond milliseconds [3].
>
> This leads to odd cases: CAST(v AS TIME) would fail while CAST(CAST(v AS
> STRING) AS TIME) succeeds on the same value. And the sibling case is
> undefined in the FLIP: what does CAST(variant AS TIMESTAMP(3)) do when the
> variant holds a nanosecond timestamp. Truncate like the existing rule, or
> faillike the new TIME rule?
>
> I'd suggest truncating for consistency, and treating FLINK-39214 [3] as the
> proper long-term fix that removes the TIME gap entirely.
>
> Best,
> Mate
>
> [1]
>
> https://github.com/apache/flink/blob/33a198ae41ca1f11a8a8f313a5295551d1d591df/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/TimestampToTimestampCastRule.java#L95
> [2]
>
> https://github.com/apache/flink/blob/33a198ae41ca1f11a8a8f313a5295551d1d591df/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java#L500
> [3] https://issues.apache.org/jira/browse/FLINK-39214
>
>
> On Tue, Aug 4, 2026 at 4:22 PM Timo Walther <tw...@apache.org> wrote:
>
> > Thanks for the explanation Ramin. LGTM
> >
> > Cheers,
> > Timo
> >
> > On 04.08.26 15:27, Ramin Gharib wrote:
> > > Hello Timo,
> > > Thanks for looking into this and the feedback.
> > >
> > > Yes, for SQL: a Variant TIME lands as TIME(3), and per the FLIP a
> > sub-millisecond value fails the cast instead of truncating.
> > > Small correction on the DataStream side: the ceiling for TIME is
> micros,
> > not nanos. Variant code 17 is TIME(isAdjustedToUTC=false, MICROS) and the
> > spec has no nanosecond TIME, so getTime() surfaces at most precision 6.
> > >
> > > Nanosecond precision (9) applies to codes 18 and 19, the nanosecond
> > timestamps, not to TIME. TIMESTAMP(9) and TIMESTAMP_LTZ(9) already cover
> > them, so both SQL and DataStream keep the full nanoseconds. TIME is the
> > only code with a gap, since its Variant precision is micros (6) while SQL
> > TIME caps at millis (3).
> > >
> > > I've updated the FLIP to make this explicit (the TIME(3) mapping in the
> > table plus a note on the SQL-vs-flink-core precision).
> > >
> > > Best,
> > >
> > > Ramin
> > >
> > > From: Timo Walther <tw...@apache.org>
> > > Date: Tuesday, 4. August 2026 at 12:59
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage
> > with UUID and Timestamps
> > >
> > > Hi Ramin,
> > >
> > > thank you for evolving the VARIANT type further. I fully support the
> > FLIP.
> > >
> > > The ask for a UUID data type came up a couple of times in the past
> > > because UUIDs are in almost every data pipeline.
> > >
> > > Also full timestamp and time support incl nano second precision is a
> > > topic that always pops up and support for it avoids friction.
> > >
> > > One question the statement: "time without time zone (us)"
> > >
> > > Since Flink SQL only supports TIME(3), I guess we limit VARIANT also to
> > > millis within the SQL runtime, right? For DataStream API we can support
> > > 9, right?
> > >
> > > Cheers,
> > > Timo
> > >
> > > On 04.08.26 12:28, Ramin Gharib wrote:
> > >> Hi everyone,
> > >>
> > >> I'd like to start a discussion on FLIP-604: Complete VARIANT
> > >> Primitive Coverage with UUID and Timestamps. [1]
> > >>
> > >> FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements
> > Parquet
> > >> Variant primitive codes 0 through 16. The open Variant spec goes
> > >> through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
> > >> written by Spark, Iceberg, or any Parquet writer that holds one of
> > >> these values cannot be read by Flink today, which breaks the
> > >> lakehouse round-trip VARIANT was meant to support.
> > >>
> > >> This FLIP completes the primitive set. TIME and the nanosecond
> > >> timestamps reuse types Flink already has. UUID needs a type to land
> > >> on, so the FLIP also adds a native 16-byte UUID type as a first-class
> > >> SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
> > >> UUID_V7 generator functions.
> > >>
> > >> One point worth noting: no Calcite upgrade is required. Calcite has
> > >> had a UUID type since 1.39, the same release FLIP-521 adopted for
> > >> VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
> > >> matches every engine with a native UUID type, including PostgreSQL,
> > >> Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
> > >>
> > >> Looking forward to your feedback.
> > >>
> > >> Best,
> > >> Ramin Gharib
> > >>
> > >> [1]
> >
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
> > >> [2]
> >
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
> > >>
> > >>
> > >
> > >
> >
> >
>
Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Posted by Mate Czagany <cz...@gmail.com>.
Hi Ramin,
Thank you for working on this. Looks good overall, one question on the cast
semantics for VARIANT TIME.
The FLIP says CAST(varant AS TIME) fails when the value has sub-millisecond
digits. I think that would make VARIANT the only cast where narrowing the
precision fails instead of truncating. Everywhere else Flink silently
truncates: TIMESTAMP(9) -> TIMESTAMP(3) truncates [1],
CAST('12:34:56.123456789' AS TIME) drops the fraction [2]. In CastRulesTest
there is also a reference to an improvement ticket to support TIME type
precision beyond milliseconds [3].
This leads to odd cases: CAST(v AS TIME) would fail while CAST(CAST(v AS
STRING) AS TIME) succeeds on the same value. And the sibling case is
undefined in the FLIP: what does CAST(variant AS TIMESTAMP(3)) do when the
variant holds a nanosecond timestamp. Truncate like the existing rule, or
faillike the new TIME rule?
I'd suggest truncating for consistency, and treating FLINK-39214 [3] as the
proper long-term fix that removes the TIME gap entirely.
Best,
Mate
[1]
https://github.com/apache/flink/blob/33a198ae41ca1f11a8a8f313a5295551d1d591df/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/TimestampToTimestampCastRule.java#L95
[2]
https://github.com/apache/flink/blob/33a198ae41ca1f11a8a8f313a5295551d1d591df/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java#L500
[3] https://issues.apache.org/jira/browse/FLINK-39214
On Tue, Aug 4, 2026 at 4:22 PM Timo Walther <tw...@apache.org> wrote:
> Thanks for the explanation Ramin. LGTM
>
> Cheers,
> Timo
>
> On 04.08.26 15:27, Ramin Gharib wrote:
> > Hello Timo,
> > Thanks for looking into this and the feedback.
> >
> > Yes, for SQL: a Variant TIME lands as TIME(3), and per the FLIP a
> sub-millisecond value fails the cast instead of truncating.
> > Small correction on the DataStream side: the ceiling for TIME is micros,
> not nanos. Variant code 17 is TIME(isAdjustedToUTC=false, MICROS) and the
> spec has no nanosecond TIME, so getTime() surfaces at most precision 6.
> >
> > Nanosecond precision (9) applies to codes 18 and 19, the nanosecond
> timestamps, not to TIME. TIMESTAMP(9) and TIMESTAMP_LTZ(9) already cover
> them, so both SQL and DataStream keep the full nanoseconds. TIME is the
> only code with a gap, since its Variant precision is micros (6) while SQL
> TIME caps at millis (3).
> >
> > I've updated the FLIP to make this explicit (the TIME(3) mapping in the
> table plus a note on the SQL-vs-flink-core precision).
> >
> > Best,
> >
> > Ramin
> >
> > From: Timo Walther <tw...@apache.org>
> > Date: Tuesday, 4. August 2026 at 12:59
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage
> with UUID and Timestamps
> >
> > Hi Ramin,
> >
> > thank you for evolving the VARIANT type further. I fully support the
> FLIP.
> >
> > The ask for a UUID data type came up a couple of times in the past
> > because UUIDs are in almost every data pipeline.
> >
> > Also full timestamp and time support incl nano second precision is a
> > topic that always pops up and support for it avoids friction.
> >
> > One question the statement: "time without time zone (us)"
> >
> > Since Flink SQL only supports TIME(3), I guess we limit VARIANT also to
> > millis within the SQL runtime, right? For DataStream API we can support
> > 9, right?
> >
> > Cheers,
> > Timo
> >
> > On 04.08.26 12:28, Ramin Gharib wrote:
> >> Hi everyone,
> >>
> >> I'd like to start a discussion on FLIP-604: Complete VARIANT
> >> Primitive Coverage with UUID and Timestamps. [1]
> >>
> >> FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements
> Parquet
> >> Variant primitive codes 0 through 16. The open Variant spec goes
> >> through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
> >> written by Spark, Iceberg, or any Parquet writer that holds one of
> >> these values cannot be read by Flink today, which breaks the
> >> lakehouse round-trip VARIANT was meant to support.
> >>
> >> This FLIP completes the primitive set. TIME and the nanosecond
> >> timestamps reuse types Flink already has. UUID needs a type to land
> >> on, so the FLIP also adds a native 16-byte UUID type as a first-class
> >> SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
> >> UUID_V7 generator functions.
> >>
> >> One point worth noting: no Calcite upgrade is required. Calcite has
> >> had a UUID type since 1.39, the same release FLIP-521 adopted for
> >> VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
> >> matches every engine with a native UUID type, including PostgreSQL,
> >> Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
> >>
> >> Looking forward to your feedback.
> >>
> >> Best,
> >> Ramin Gharib
> >>
> >> [1]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
> >> [2]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
> >>
> >>
> >
> >
>
>
Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Posted by Timo Walther <tw...@apache.org>.
Thanks for the explanation Ramin. LGTM
Cheers,
Timo
On 04.08.26 15:27, Ramin Gharib wrote:
> Hello Timo,
> Thanks for looking into this and the feedback.
>
> Yes, for SQL: a Variant TIME lands as TIME(3), and per the FLIP a sub-millisecond value fails the cast instead of truncating.
> Small correction on the DataStream side: the ceiling for TIME is micros, not nanos. Variant code 17 is TIME(isAdjustedToUTC=false, MICROS) and the spec has no nanosecond TIME, so getTime() surfaces at most precision 6.
>
> Nanosecond precision (9) applies to codes 18 and 19, the nanosecond timestamps, not to TIME. TIMESTAMP(9) and TIMESTAMP_LTZ(9) already cover them, so both SQL and DataStream keep the full nanoseconds. TIME is the only code with a gap, since its Variant precision is micros (6) while SQL TIME caps at millis (3).
>
> I've updated the FLIP to make this explicit (the TIME(3) mapping in the table plus a note on the SQL-vs-flink-core precision).
>
> Best,
>
> Ramin
>
> From: Timo Walther <tw...@apache.org>
> Date: Tuesday, 4. August 2026 at 12:59
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
>
> Hi Ramin,
>
> thank you for evolving the VARIANT type further. I fully support the FLIP.
>
> The ask for a UUID data type came up a couple of times in the past
> because UUIDs are in almost every data pipeline.
>
> Also full timestamp and time support incl nano second precision is a
> topic that always pops up and support for it avoids friction.
>
> One question the statement: "time without time zone (us)"
>
> Since Flink SQL only supports TIME(3), I guess we limit VARIANT also to
> millis within the SQL runtime, right? For DataStream API we can support
> 9, right?
>
> Cheers,
> Timo
>
> On 04.08.26 12:28, Ramin Gharib wrote:
>> Hi everyone,
>>
>> I'd like to start a discussion on FLIP-604: Complete VARIANT
>> Primitive Coverage with UUID and Timestamps. [1]
>>
>> FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements Parquet
>> Variant primitive codes 0 through 16. The open Variant spec goes
>> through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
>> written by Spark, Iceberg, or any Parquet writer that holds one of
>> these values cannot be read by Flink today, which breaks the
>> lakehouse round-trip VARIANT was meant to support.
>>
>> This FLIP completes the primitive set. TIME and the nanosecond
>> timestamps reuse types Flink already has. UUID needs a type to land
>> on, so the FLIP also adds a native 16-byte UUID type as a first-class
>> SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
>> UUID_V7 generator functions.
>>
>> One point worth noting: no Calcite upgrade is required. Calcite has
>> had a UUID type since 1.39, the same release FLIP-521 adopted for
>> VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
>> matches every engine with a native UUID type, including PostgreSQL,
>> Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
>>
>> Looking forward to your feedback.
>>
>> Best,
>> Ramin Gharib
>>
>> [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
>> [2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
>>
>>
>
>
Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Posted by Ramin Gharib <ra...@gmail.com>.
Hello Timo,
Thanks for looking into this and the feedback.
Yes, for SQL: a Variant TIME lands as TIME(3), and per the FLIP a sub-millisecond value fails the cast instead of truncating.
Small correction on the DataStream side: the ceiling for TIME is micros, not nanos. Variant code 17 is TIME(isAdjustedToUTC=false, MICROS) and the spec has no nanosecond TIME, so getTime() surfaces at most precision 6.
Nanosecond precision (9) applies to codes 18 and 19, the nanosecond timestamps, not to TIME. TIMESTAMP(9) and TIMESTAMP_LTZ(9) already cover them, so both SQL and DataStream keep the full nanoseconds. TIME is the only code with a gap, since its Variant precision is micros (6) while SQL TIME caps at millis (3).
I've updated the FLIP to make this explicit (the TIME(3) mapping in the table plus a note on the SQL-vs-flink-core precision).
Best,
Ramin
From: Timo Walther <tw...@apache.org>
Date: Tuesday, 4. August 2026 at 12:59
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Hi Ramin,
thank you for evolving the VARIANT type further. I fully support the FLIP.
The ask for a UUID data type came up a couple of times in the past
because UUIDs are in almost every data pipeline.
Also full timestamp and time support incl nano second precision is a
topic that always pops up and support for it avoids friction.
One question the statement: "time without time zone (us)"
Since Flink SQL only supports TIME(3), I guess we limit VARIANT also to
millis within the SQL runtime, right? For DataStream API we can support
9, right?
Cheers,
Timo
On 04.08.26 12:28, Ramin Gharib wrote:
> Hi everyone,
>
> I'd like to start a discussion on FLIP-604: Complete VARIANT
> Primitive Coverage with UUID and Timestamps. [1]
>
> FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements Parquet
> Variant primitive codes 0 through 16. The open Variant spec goes
> through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
> written by Spark, Iceberg, or any Parquet writer that holds one of
> these values cannot be read by Flink today, which breaks the
> lakehouse round-trip VARIANT was meant to support.
>
> This FLIP completes the primitive set. TIME and the nanosecond
> timestamps reuse types Flink already has. UUID needs a type to land
> on, so the FLIP also adds a native 16-byte UUID type as a first-class
> SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
> UUID_V7 generator functions.
>
> One point worth noting: no Calcite upgrade is required. Calcite has
> had a UUID type since 1.39, the same release FLIP-521 adopted for
> VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
> matches every engine with a native UUID type, including PostgreSQL,
> Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
>
> Looking forward to your feedback.
>
> Best,
> Ramin Gharib
>
> [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
> [2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
>
>
Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Posted by Timo Walther <tw...@apache.org>.
Hi Ramin,
thank you for evolving the VARIANT type further. I fully support the FLIP.
The ask for a UUID data type came up a couple of times in the past
because UUIDs are in almost every data pipeline.
Also full timestamp and time support incl nano second precision is a
topic that always pops up and support for it avoids friction.
One question the statement: "time without time zone (us)"
Since Flink SQL only supports TIME(3), I guess we limit VARIANT also to
millis within the SQL runtime, right? For DataStream API we can support
9, right?
Cheers,
Timo
On 04.08.26 12:28, Ramin Gharib wrote:
> Hi everyone,
>
> I'd like to start a discussion on FLIP-604: Complete VARIANT
> Primitive Coverage with UUID and Timestamps. [1]
>
> FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements Parquet
> Variant primitive codes 0 through 16. The open Variant spec goes
> through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
> written by Spark, Iceberg, or any Parquet writer that holds one of
> these values cannot be read by Flink today, which breaks the
> lakehouse round-trip VARIANT was meant to support.
>
> This FLIP completes the primitive set. TIME and the nanosecond
> timestamps reuse types Flink already has. UUID needs a type to land
> on, so the FLIP also adds a native 16-byte UUID type as a first-class
> SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
> UUID_V7 generator functions.
>
> One point worth noting: no Calcite upgrade is required. Calcite has
> had a UUID type since 1.39, the same release FLIP-521 adopted for
> VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
> matches every engine with a native UUID type, including PostgreSQL,
> Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
>
> Looking forward to your feedback.
>
> Best,
> Ramin Gharib
>
> [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
> [2] https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
>
>
Re: [DISCUSS] FLIP-604: Complete VARIANT Primitive Coverage with UUID and Timestamps
Posted by Ramin Gharib <ra...@gmail.com>.
Hi everyone,
The discussion on FLIP-604 [1] has been open for two weeks now. Thanks to
Timo, Mate, and Gustavo for the feedback. All the points raised have been
addressed and folded into the FLIP. The thread has settled and I don't see
any open items.
Given the consensus, I'll start a VOTE thread shortly.
Thanks again,
Ramin
[1]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
On Tue, Aug 4, 2026 at 12:28 PM Ramin Gharib <ra...@gmail.com> wrote:
> Hi everyone,
>
> I'd like to start a discussion on FLIP-604: Complete VARIANT
> Primitive Coverage with UUID and Timestamps. [1]
>
> FLIP-521 [2] shipped VARIANT in Flink 2.1, but it only implements Parquet
> Variant primitive codes 0 through 16. The open Variant spec goes
> through 20: TIME, nanosecond timestamps, and UUID. So a VARIANT
> written by Spark, Iceberg, or any Parquet writer that holds one of
> these values cannot be read by Flink today, which breaks the
> lakehouse round-trip VARIANT was meant to support.
>
> This FLIP completes the primitive set. TIME and the nanosecond
> timestamps reuse types Flink already has. UUID needs a type to land
> on, so the FLIP also adds a native 16-byte UUID type as a first-class
> SQL type, with casts, ordering, a UUID literal, and UUID_V4 and
> UUID_V7 generator functions.
>
> One point worth noting: no Calcite upgrade is required. Calcite has
> had a UUID type since 1.39, the same release FLIP-521 adopted for
> VARIANT, and Flink already runs Calcite 1.41. The 16-byte layout also
> matches every engine with a native UUID type, including PostgreSQL,
> Trino, DuckDB, ClickHouse, Snowflake, and Iceberg.
>
> Looking forward to your feedback.
>
> Best,
> Ramin Gharib
>
> [1]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/446070993/FLIP-604+Complete+VARIANT+Primitive+Coverage+with+UUID+and+Timestamps
> [2]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/349637099/FLIP-521+Integrating+Variant+Type+into+Flink+Enabling+Efficient+Semi-Structured+Data+Processing
>
>