You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Xuannan Su <su...@gmail.com> on 2025/04/14 10:10:00 UTC
[DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Hi devs,
I’d like to start a discussion around FLIP-521: Integrating Variant
Type into Flink: Enabling Efficient Semi-Structured Data
Processing[1]. Working with semi-structured data has long been a
foundational scenario of the Lakehouse. While JSON has traditionally
served as the primary storage format for such data, its implementation
as serialized strings introduces significant inefficiencies.
In this FLIP, we integrate the Variant encoding, which is a compact
binary representation of semi-structured data[2], to improve the
performance of processing semi-structured data. As Paimon has
supported the Variant type recently[3], this FLIP would allow Flink to
further leverage Paimon's storage-layer optimizations, improving
performance and resource utilization for semi-structured data
pipelines.
Best,
Xuannan
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
[2] https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
[3] https://github.com/apache/paimon/issues/4471
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by jk liu <jk...@gmail.com>.
Hi Xuannan,
First , I’m not sure whether discussing this kind of detailed topic here is
too late, or whether there is a better place/channel for it. If this is not
the right place, I’m sorry—and I’d appreciate it if you could point me to
where this should be discussed.
I noticed that FLIP-521
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=349637099#FLIP521:IntegratingVariantTypeintoFlink:EnablingEfficientSemiStructuredDataProcessing-IntroduceBuilt-inFunctiontoAccessfieldorelementintheVariant>
introduces
an SQL function to access variant.field (and the corresponding table
function seems to be VARIANT.get(field)—I’m not sure if that is a typo).
However, in the SQL standard and in similar projects (e.g., Spark and
Calcite), .field is typically used to access data types with a fixed
schema, such as ROW or STRUCT. So I’m concerned that introducing .field access
for VARIANT may not be appropriate.
If I’m misunderstanding anything, I’d be very happy to be corrected.
Best,
Jinkun
Xuannan Su <su...@gmail.com> 于2025年7月2日周三 16:17写道：
> Hi Alexey,
>
> Thanks for the reminder! The FLIP is updated.
>
> Best,
> Xuannan
>
>
> > On Jul 2, 2025, at 08:02, Alexey Leonov-Vendrovskiy <
> vendrovsky@gmail.com> wrote:
> >
> > Hi Xuannan,
> > Can you link the voting thread and add a JIRA link to the FLIP doc?
> >
> > Thanks,
> > Alexey
> >
> > On Wed, May 7, 2025 at 12:28 AM Xuannan Su <su...@gmail.com>
> wrote:
> >
> >> Hi Timo,
> >>
> >> Thanks for the suggestion!
> >>
> >> I updated the FLIP to use `isPrimitive()` and `VariantTypeException`.
> >>
> >> Regarding `JSON()` and `PARSE_JSON()`, we should keep the return type
> >> of the `JSON()` method as it is, and use the `PARSE_JSON()` method to
> >> convert a JSON string to a variant type. `JSON`, `JSON_OBJECT`,
> >> `JSON_ARRAY` are a set of methods to build a JSON string, while
> >> `PARSE_JSON` converts a JSON string to a variant data type, which can
> >> work well together and should not cause confusion.
> >>
> >> You are right that casting the variant to the STRUCTURED and ROW types
> >> is useful. It should be included in the FLIP, while we may not support
> >> it in the first version. The FLIP is updated.
> >>
> >> Best,
> >> Xuannan
> >>
> >>
> >>
> >> On Mon, May 5, 2025 at 9:54 PM Timo Walther <tw...@apache.org> wrote:
> >>>
> >>> Hi Xuannan,
> >>>
> >>> thank you for updating the FLIP and addressing my comments. The FLIP is
> >>> in a very good shape now, +1 for voting. I just found some last
> >>> remaining items that would be good to be addressed in the FLIP. But not
> >>> blocker for the voting process:
> >>>
> >>>> isScalar()
> >>>
> >>> I thought about the naming again. I would rather vote for
> >>> `isPrimitive()`. Our ScalarFunctions are also able to take and return
> >>> arrays and maps. An `isPrimitive()` should avoid this confusion.
> >>>
> >>>> VariantUnexpectedTypeException
> >>>
> >>> Rather call it VariantTypeException? Shorter is better. Otherwise a
> >>> UnexpectedVariantTypeException would read better. But a general
> >>> VariantTypeException should be good enough.
> >>>
> >>>> I cannot figure out how it can work with the current JSON_OBJECT and
> >>> JSON_ARRAY functions
> >>>
> >>> We left the return type of JSON() open to future work. JSON() can only
> >>> be used directly as parameters for values in JSON_OBJECT and
> JSON_ARRAY,
> >>> because we didn't have a JSON type so far. So JSON() can return a
> >>> VARIANT type that is directly understood by JSON_OBJECT and JSON_ARRAY
> >>> functions.
> >>> I'm fine with PARSE_JSON as well, but we should double check the
> >>> semantics with JSON() to avoid any kind of confusion.
> >>>
> >>>> Therefore, instead of going for a non-trivial temporary solution
> >>>> that will be removed later, we are better off justbumping the
> >>>> Calcite version, which is inevitable.
> >>>
> >>> Great to hear that we aim to update Calcite. Happy to support reviewing
> >>> PRs that bump Calcite versions. I'm sure Sergey can help here as well.
> >>> We should try to perform this update in stages to avoid introducing
> >>> regressions.
> >>>
> >>>> Only a scalar value Variant is allowed to be cast to the
> >>>> corresponding type
> >>>
> >>> Can you elaborate on this? It could be quite useful for the Table API
> >>> integration if we could supporting casting to STRUCTURED types and thus
> >>> POJOs in the future. This doesn't have to be supported in version 1 but
> >>> could be useful. For rows and structured types, we support nested
> >>> casting when calling functions.
> >>>
> >>> Looking forward to this feature!
> >>>
> >>> Regards,
> >>> Timo
> >>>
> >>>
> >>> On 03.05.25 01:32, Xuannan Su wrote:
> >>>> Hi everyone,
> >>>>
> >>>> Thank you for all the comments! Please let me know if you have further
> >>>> comments. If there are no further comments, I'd
> >>>> like to close the discussion and start the voting in a few days.
> >>>>
> >>>> Best,
> >>>> Xuannan
> >>>>
> >>>>
> >>>> On Thu, May 1, 2025 at 20:21 Xuannan Su <su...@gmail.com>
> wrote:
> >>>>
> >>>>> Hi Aihua,
> >>>>>
> >>>>> We planning to support Variant shredding soon after we support the
> >>>>> basis variant encoding as stated in the Future Work section of the
> >>>>> FLIP.
> >>>>>
> >>>>> Best,
> >>>>> Xuannan
> >>>>>
> >>>>> On Thu, May 1, 2025 at 1:43 PM Aihua Xu <ai...@apache.org> wrote:
> >>>>>>
> >>>>>> Hi Xuannan,
> >>>>>>
> >>>>>> Are we planning to support Variant shredding in the future or just
> >> basic
> >>>>> variant encoding for Flink?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Aihua
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xuannan Su <su...@gmail.com>.
Hi Alexey,
Thanks for the reminder! The FLIP is updated.
Best,
Xuannan
> On Jul 2, 2025, at 08:02, Alexey Leonov-Vendrovskiy <ve...@gmail.com> wrote:
>
> Hi Xuannan,
> Can you link the voting thread and add a JIRA link to the FLIP doc?
>
> Thanks,
> Alexey
>
> On Wed, May 7, 2025 at 12:28 AM Xuannan Su <su...@gmail.com> wrote:
>
>> Hi Timo,
>>
>> Thanks for the suggestion!
>>
>> I updated the FLIP to use `isPrimitive()` and `VariantTypeException`.
>>
>> Regarding `JSON()` and `PARSE_JSON()`, we should keep the return type
>> of the `JSON()` method as it is, and use the `PARSE_JSON()` method to
>> convert a JSON string to a variant type. `JSON`, `JSON_OBJECT`,
>> `JSON_ARRAY` are a set of methods to build a JSON string, while
>> `PARSE_JSON` converts a JSON string to a variant data type, which can
>> work well together and should not cause confusion.
>>
>> You are right that casting the variant to the STRUCTURED and ROW types
>> is useful. It should be included in the FLIP, while we may not support
>> it in the first version. The FLIP is updated.
>>
>> Best,
>> Xuannan
>>
>>
>>
>> On Mon, May 5, 2025 at 9:54 PM Timo Walther <tw...@apache.org> wrote:
>>>
>>> Hi Xuannan,
>>>
>>> thank you for updating the FLIP and addressing my comments. The FLIP is
>>> in a very good shape now, +1 for voting. I just found some last
>>> remaining items that would be good to be addressed in the FLIP. But not
>>> blocker for the voting process:
>>>
>>>> isScalar()
>>>
>>> I thought about the naming again. I would rather vote for
>>> `isPrimitive()`. Our ScalarFunctions are also able to take and return
>>> arrays and maps. An `isPrimitive()` should avoid this confusion.
>>>
>>>> VariantUnexpectedTypeException
>>>
>>> Rather call it VariantTypeException? Shorter is better. Otherwise a
>>> UnexpectedVariantTypeException would read better. But a general
>>> VariantTypeException should be good enough.
>>>
>>>> I cannot figure out how it can work with the current JSON_OBJECT and
>>> JSON_ARRAY functions
>>>
>>> We left the return type of JSON() open to future work. JSON() can only
>>> be used directly as parameters for values in JSON_OBJECT and JSON_ARRAY,
>>> because we didn't have a JSON type so far. So JSON() can return a
>>> VARIANT type that is directly understood by JSON_OBJECT and JSON_ARRAY
>>> functions.
>>> I'm fine with PARSE_JSON as well, but we should double check the
>>> semantics with JSON() to avoid any kind of confusion.
>>>
>>>> Therefore, instead of going for a non-trivial temporary solution
>>>> that will be removed later, we are better off justbumping the
>>>> Calcite version, which is inevitable.
>>>
>>> Great to hear that we aim to update Calcite. Happy to support reviewing
>>> PRs that bump Calcite versions. I'm sure Sergey can help here as well.
>>> We should try to perform this update in stages to avoid introducing
>>> regressions.
>>>
>>>> Only a scalar value Variant is allowed to be cast to the
>>>> corresponding type
>>>
>>> Can you elaborate on this? It could be quite useful for the Table API
>>> integration if we could supporting casting to STRUCTURED types and thus
>>> POJOs in the future. This doesn't have to be supported in version 1 but
>>> could be useful. For rows and structured types, we support nested
>>> casting when calling functions.
>>>
>>> Looking forward to this feature!
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> On 03.05.25 01:32, Xuannan Su wrote:
>>>> Hi everyone,
>>>>
>>>> Thank you for all the comments! Please let me know if you have further
>>>> comments. If there are no further comments, I'd
>>>> like to close the discussion and start the voting in a few days.
>>>>
>>>> Best,
>>>> Xuannan
>>>>
>>>>
>>>> On Thu, May 1, 2025 at 20:21 Xuannan Su <su...@gmail.com> wrote:
>>>>
>>>>> Hi Aihua,
>>>>>
>>>>> We planning to support Variant shredding soon after we support the
>>>>> basis variant encoding as stated in the Future Work section of the
>>>>> FLIP.
>>>>>
>>>>> Best,
>>>>> Xuannan
>>>>>
>>>>> On Thu, May 1, 2025 at 1:43 PM Aihua Xu <ai...@apache.org> wrote:
>>>>>>
>>>>>> Hi Xuannan,
>>>>>>
>>>>>> Are we planning to support Variant shredding in the future or just
>> basic
>>>>> variant encoding for Flink?
>>>>>>
>>>>>> Thanks,
>>>>>> Aihua
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Alexey Leonov-Vendrovskiy <ve...@gmail.com>.
Hi Xuannan,
Can you link the voting thread and add a JIRA link to the FLIP doc?
Thanks,
Alexey
On Wed, May 7, 2025 at 12:28 AM Xuannan Su <su...@gmail.com> wrote:
> Hi Timo,
>
> Thanks for the suggestion!
>
> I updated the FLIP to use `isPrimitive()` and `VariantTypeException`.
>
> Regarding `JSON()` and `PARSE_JSON()`, we should keep the return type
> of the `JSON()` method as it is, and use the `PARSE_JSON()` method to
> convert a JSON string to a variant type. `JSON`, `JSON_OBJECT`,
> `JSON_ARRAY` are a set of methods to build a JSON string, while
> `PARSE_JSON` converts a JSON string to a variant data type, which can
> work well together and should not cause confusion.
>
> You are right that casting the variant to the STRUCTURED and ROW types
> is useful. It should be included in the FLIP, while we may not support
> it in the first version. The FLIP is updated.
>
> Best,
> Xuannan
>
>
>
> On Mon, May 5, 2025 at 9:54 PM Timo Walther <tw...@apache.org> wrote:
> >
> > Hi Xuannan,
> >
> > thank you for updating the FLIP and addressing my comments. The FLIP is
> > in a very good shape now, +1 for voting. I just found some last
> > remaining items that would be good to be addressed in the FLIP. But not
> > blocker for the voting process:
> >
> > > isScalar()
> >
> > I thought about the naming again. I would rather vote for
> > `isPrimitive()`. Our ScalarFunctions are also able to take and return
> > arrays and maps. An `isPrimitive()` should avoid this confusion.
> >
> > > VariantUnexpectedTypeException
> >
> > Rather call it VariantTypeException? Shorter is better. Otherwise a
> > UnexpectedVariantTypeException would read better. But a general
> > VariantTypeException should be good enough.
> >
> > > I cannot figure out how it can work with the current JSON_OBJECT and
> > JSON_ARRAY functions
> >
> > We left the return type of JSON() open to future work. JSON() can only
> > be used directly as parameters for values in JSON_OBJECT and JSON_ARRAY,
> > because we didn't have a JSON type so far. So JSON() can return a
> > VARIANT type that is directly understood by JSON_OBJECT and JSON_ARRAY
> > functions.
> > I'm fine with PARSE_JSON as well, but we should double check the
> > semantics with JSON() to avoid any kind of confusion.
> >
> > > Therefore, instead of going for a non-trivial temporary solution
> > > that will be removed later, we are better off justbumping the
> > > Calcite version, which is inevitable.
> >
> > Great to hear that we aim to update Calcite. Happy to support reviewing
> > PRs that bump Calcite versions. I'm sure Sergey can help here as well.
> > We should try to perform this update in stages to avoid introducing
> > regressions.
> >
> > > Only a scalar value Variant is allowed to be cast to the
> > > corresponding type
> >
> > Can you elaborate on this? It could be quite useful for the Table API
> > integration if we could supporting casting to STRUCTURED types and thus
> > POJOs in the future. This doesn't have to be supported in version 1 but
> > could be useful. For rows and structured types, we support nested
> > casting when calling functions.
> >
> > Looking forward to this feature!
> >
> > Regards,
> > Timo
> >
> >
> > On 03.05.25 01:32, Xuannan Su wrote:
> > > Hi everyone,
> > >
> > > Thank you for all the comments! Please let me know if you have further
> > > comments. If there are no further comments, I'd
> > > like to close the discussion and start the voting in a few days.
> > >
> > > Best,
> > > Xuannan
> > >
> > >
> > > On Thu, May 1, 2025 at 20:21 Xuannan Su <su...@gmail.com> wrote:
> > >
> > >> Hi Aihua,
> > >>
> > >> We planning to support Variant shredding soon after we support the
> > >> basis variant encoding as stated in the Future Work section of the
> > >> FLIP.
> > >>
> > >> Best,
> > >> Xuannan
> > >>
> > >> On Thu, May 1, 2025 at 1:43 PM Aihua Xu <ai...@apache.org> wrote:
> > >>>
> > >>> Hi Xuannan,
> > >>>
> > >>> Are we planning to support Variant shredding in the future or just
> basic
> > >> variant encoding for Flink?
> > >>>
> > >>> Thanks,
> > >>> Aihua
> > >>
> > >
> >
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xuannan Su <su...@gmail.com>.
Hi Timo,
Thanks for the suggestion!
I updated the FLIP to use `isPrimitive()` and `VariantTypeException`.
Regarding `JSON()` and `PARSE_JSON()`, we should keep the return type
of the `JSON()` method as it is, and use the `PARSE_JSON()` method to
convert a JSON string to a variant type. `JSON`, `JSON_OBJECT`,
`JSON_ARRAY` are a set of methods to build a JSON string, while
`PARSE_JSON` converts a JSON string to a variant data type, which can
work well together and should not cause confusion.
You are right that casting the variant to the STRUCTURED and ROW types
is useful. It should be included in the FLIP, while we may not support
it in the first version. The FLIP is updated.
Best,
Xuannan
On Mon, May 5, 2025 at 9:54 PM Timo Walther <tw...@apache.org> wrote:
>
> Hi Xuannan,
>
> thank you for updating the FLIP and addressing my comments. The FLIP is
> in a very good shape now, +1 for voting. I just found some last
> remaining items that would be good to be addressed in the FLIP. But not
> blocker for the voting process:
>
> > isScalar()
>
> I thought about the naming again. I would rather vote for
> `isPrimitive()`. Our ScalarFunctions are also able to take and return
> arrays and maps. An `isPrimitive()` should avoid this confusion.
>
> > VariantUnexpectedTypeException
>
> Rather call it VariantTypeException? Shorter is better. Otherwise a
> UnexpectedVariantTypeException would read better. But a general
> VariantTypeException should be good enough.
>
> > I cannot figure out how it can work with the current JSON_OBJECT and
> JSON_ARRAY functions
>
> We left the return type of JSON() open to future work. JSON() can only
> be used directly as parameters for values in JSON_OBJECT and JSON_ARRAY,
> because we didn't have a JSON type so far. So JSON() can return a
> VARIANT type that is directly understood by JSON_OBJECT and JSON_ARRAY
> functions.
> I'm fine with PARSE_JSON as well, but we should double check the
> semantics with JSON() to avoid any kind of confusion.
>
> > Therefore, instead of going for a non-trivial temporary solution
> > that will be removed later, we are better off justbumping the
> > Calcite version, which is inevitable.
>
> Great to hear that we aim to update Calcite. Happy to support reviewing
> PRs that bump Calcite versions. I'm sure Sergey can help here as well.
> We should try to perform this update in stages to avoid introducing
> regressions.
>
> > Only a scalar value Variant is allowed to be cast to the
> > corresponding type
>
> Can you elaborate on this? It could be quite useful for the Table API
> integration if we could supporting casting to STRUCTURED types and thus
> POJOs in the future. This doesn't have to be supported in version 1 but
> could be useful. For rows and structured types, we support nested
> casting when calling functions.
>
> Looking forward to this feature!
>
> Regards,
> Timo
>
>
> On 03.05.25 01:32, Xuannan Su wrote:
> > Hi everyone,
> >
> > Thank you for all the comments! Please let me know if you have further
> > comments. If there are no further comments, I'd
> > like to close the discussion and start the voting in a few days.
> >
> > Best,
> > Xuannan
> >
> >
> > On Thu, May 1, 2025 at 20:21 Xuannan Su <su...@gmail.com> wrote:
> >
> >> Hi Aihua,
> >>
> >> We planning to support Variant shredding soon after we support the
> >> basis variant encoding as stated in the Future Work section of the
> >> FLIP.
> >>
> >> Best,
> >> Xuannan
> >>
> >> On Thu, May 1, 2025 at 1:43 PM Aihua Xu <ai...@apache.org> wrote:
> >>>
> >>> Hi Xuannan,
> >>>
> >>> Are we planning to support Variant shredding in the future or just basic
> >> variant encoding for Flink?
> >>>
> >>> Thanks,
> >>> Aihua
> >>
> >
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Timo Walther <tw...@apache.org>.
Hi Xuannan,
thank you for updating the FLIP and addressing my comments. The FLIP is
in a very good shape now, +1 for voting. I just found some last
remaining items that would be good to be addressed in the FLIP. But not
blocker for the voting process:
> isScalar()
I thought about the naming again. I would rather vote for
`isPrimitive()`. Our ScalarFunctions are also able to take and return
arrays and maps. An `isPrimitive()` should avoid this confusion.
> VariantUnexpectedTypeException
Rather call it VariantTypeException? Shorter is better. Otherwise a
UnexpectedVariantTypeException would read better. But a general
VariantTypeException should be good enough.
> I cannot figure out how it can work with the current JSON_OBJECT and
JSON_ARRAY functions
We left the return type of JSON() open to future work. JSON() can only
be used directly as parameters for values in JSON_OBJECT and JSON_ARRAY,
because we didn't have a JSON type so far. So JSON() can return a
VARIANT type that is directly understood by JSON_OBJECT and JSON_ARRAY
functions.
I'm fine with PARSE_JSON as well, but we should double check the
semantics with JSON() to avoid any kind of confusion.
> Therefore, instead of going for a non-trivial temporary solution
> that will be removed later, we are better off justbumping the
> Calcite version, which is inevitable.
Great to hear that we aim to update Calcite. Happy to support reviewing
PRs that bump Calcite versions. I'm sure Sergey can help here as well.
We should try to perform this update in stages to avoid introducing
regressions.
> Only a scalar value Variant is allowed to be cast to the
> corresponding type
Can you elaborate on this? It could be quite useful for the Table API
integration if we could supporting casting to STRUCTURED types and thus
POJOs in the future. This doesn't have to be supported in version 1 but
could be useful. For rows and structured types, we support nested
casting when calling functions.
Looking forward to this feature!
Regards,
Timo
On 03.05.25 01:32, Xuannan Su wrote:
> Hi everyone,
>
> Thank you for all the comments! Please let me know if you have further
> comments. If there are no further comments, I'd
> like to close the discussion and start the voting in a few days.
>
> Best,
> Xuannan
>
>
> On Thu, May 1, 2025 at 20:21 Xuannan Su <su...@gmail.com> wrote:
>
>> Hi Aihua,
>>
>> We planning to support Variant shredding soon after we support the
>> basis variant encoding as stated in the Future Work section of the
>> FLIP.
>>
>> Best,
>> Xuannan
>>
>> On Thu, May 1, 2025 at 1:43 PM Aihua Xu <ai...@apache.org> wrote:
>>>
>>> Hi Xuannan,
>>>
>>> Are we planning to support Variant shredding in the future or just basic
>> variant encoding for Flink?
>>>
>>> Thanks,
>>> Aihua
>>
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xuannan Su <su...@gmail.com>.
Hi everyone,
Thank you for all the comments! Please let me know if you have further
comments. If there are no further comments, I'd
like to close the discussion and start the voting in a few days.
Best,
Xuannan
On Thu, May 1, 2025 at 20:21 Xuannan Su <su...@gmail.com> wrote:
> Hi Aihua,
>
> We planning to support Variant shredding soon after we support the
> basis variant encoding as stated in the Future Work section of the
> FLIP.
>
> Best,
> Xuannan
>
> On Thu, May 1, 2025 at 1:43 PM Aihua Xu <ai...@apache.org> wrote:
> >
> > Hi Xuannan,
> >
> > Are we planning to support Variant shredding in the future or just basic
> variant encoding for Flink?
> >
> > Thanks,
> > Aihua
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xuannan Su <su...@gmail.com>.
Hi Aihua,
We planning to support Variant shredding soon after we support the
basis variant encoding as stated in the Future Work section of the
FLIP.
Best,
Xuannan
On Thu, May 1, 2025 at 1:43 PM Aihua Xu <ai...@apache.org> wrote:
>
> Hi Xuannan,
>
> Are we planning to support Variant shredding in the future or just basic variant encoding for Flink?
>
> Thanks,
> Aihua
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Aihua Xu <ai...@apache.org>.
Hi Xuannan,
Are we planning to support Variant shredding in the future or just basic variant encoding for Flink?
Thanks,
Aihua
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xuannan Su <su...@gmail.com>.
Hi David,
Thanks for the feedback!
1. I think both ways can express the same semantics. I am just
following the API design from the `RowData`, where we have a method to
check null and return a primitive type.
2. It is indeed confusing, as the words Object and Map are used
interchangeably in the FLIP. An object typed Variant is the same as a
Map from key to variant. Because we don't have the notion of object in
the SQL type, therefore MAP makes more sense to describe the Variant
type.
3, 4. I am not sure if I understand your question. What do you mean by
json_object would be returned? I don't think that we have a
json_object type. If I understand correctly, JSON_OBJECT just returns
the json string. It doesn't make much sense that the PARSE_JSON accept
a Json string and return the same json string.
Best,
Xuannan
On Fri, Apr 25, 2025 at 10:14 PM David Radley <da...@uk.ibm.com> wrote:
>
> Hi Xuannan,
> This looks like a good addition.
>
>
> 1. I was wondering whether it is possible to have a type, but the value be null – for example a null value in a Float type and tolerate nulls being returned for float getFloat(). If so then maybe we should return an object Float instead.
> 2. You mention maps in the Flip text but do not have it has a type. I wondered what your thinking is.
> 3. In the new functions PARSE_JSON and TRY_PARSE_JSON, the text says they parse to a variant. As we support JSON_OBJECT as well, there could be an expectation that json_object would be the expected return type. Maybe we could allow the user to choose what gets returned?
> 4. Can variants be turned into json_objects and vice versa.
>
> Kind regards, David.
>
> From: Xuannan Su <su...@gmail.com>
> Date: Friday, 25 April 2025 at 12:47
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
> Hi everyone,
>
> Thank you for all the comments! If there are no further comments, I'd
> like to close the discussion and start the voting next Monday.
>
> Best,
> Xuannan
>
> On Fri, Apr 25, 2025 at 7:41 PM Lincoln Lee <li...@gmail.com> wrote:
> >
> > +1 for this FLIP. VARIANT type support will be a great addition to sql.
> > Look forward to the detailed design of the subsequent shredding
> > optimizations.
> >
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Timo Walther <tw...@apache.org> 于2025年4月22日周二 16:51写道：
> >
> > > +1 for this feature. Having a VARIANT type makes a lot of sense and
> > > together with an OBJECT type will make semi-structured data processing
> > > in Flink easier.
> > >
> > > Currently, I'm catching up with notifications after the easter holidays,
> > > but happy to give some feedback by tomorrow or Thursday as well.
> > >
> > > Thanks,
> > > Timo
> > >
> > > On 22.04.25 10:40, Jingsong Li wrote:
> > > > Thanks Xuannan for driving this discussion.
> > > >
> > > > At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
> > > > etc. are all designing and developing around Variant, and our Flink
> > > > support for Variant is very valuable.
> > > >
> > > > After a rough look at the design, there is no overall problem. It is
> > > > designed around Parquet's Variant standard, which is similar to the
> > > > overall design of Spark SQL.
> > > >
> > > > +1 for this.
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com>
> > > wrote:
> > > >>
> > > >> Hi devs,
> > > >>
> > > >> I’d like to start a discussion around FLIP-521: Integrating Variant
> > > >> Type into Flink: Enabling Efficient Semi-Structured Data
> > > >> Processing[1]. Working with semi-structured data has long been a
> > > >> foundational scenario of the Lakehouse. While JSON has traditionally
> > > >> served as the primary storage format for such data, its implementation
> > > >> as serialized strings introduces significant inefficiencies.
> > > >>
> > > >> In this FLIP, we integrate the Variant encoding, which is a compact
> > > >> binary representation of semi-structured data[2], to improve the
> > > >> performance of processing semi-structured data. As Paimon has
> > > >> supported the Variant type recently[3], this FLIP would allow Flink to
> > > >> further leverage Paimon's storage-layer optimizations, improving
> > > >> performance and resource utilization for semi-structured data
> > > >> pipelines.
> > > >>
> > > >> Best,
> > > >> Xuannan
> > > >>
> > > >> [1]
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
> > > >> [2]
> > > https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
> > > >> [3] https://github.com/apache/paimon/issues/4471
> > > >
> > >
> > >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
RE: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by David Radley <da...@uk.ibm.com>.
Hi Xuannan,
This looks like a good addition.
1. I was wondering whether it is possible to have a type, but the value be null – for example a null value in a Float type and tolerate nulls being returned for float getFloat(). If so then maybe we should return an object Float instead.
2. You mention maps in the Flip text but do not have it has a type. I wondered what your thinking is.
3. In the new functions PARSE_JSON and TRY_PARSE_JSON, the text says they parse to a variant. As we support JSON_OBJECT as well, there could be an expectation that json_object would be the expected return type. Maybe we could allow the user to choose what gets returned?
4. Can variants be turned into json_objects and vice versa.
Kind regards, David.
From: Xuannan Su <su...@gmail.com>
Date: Friday, 25 April 2025 at 12:47
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Hi everyone,
Thank you for all the comments! If there are no further comments, I'd
like to close the discussion and start the voting next Monday.
Best,
Xuannan
On Fri, Apr 25, 2025 at 7:41 PM Lincoln Lee <li...@gmail.com> wrote:
>
> +1 for this FLIP. VARIANT type support will be a great addition to sql.
> Look forward to the detailed design of the subsequent shredding
> optimizations.
>
>
> Best,
> Lincoln Lee
>
>
> Timo Walther <tw...@apache.org> 于2025年4月22日周二 16:51写道：
>
> > +1 for this feature. Having a VARIANT type makes a lot of sense and
> > together with an OBJECT type will make semi-structured data processing
> > in Flink easier.
> >
> > Currently, I'm catching up with notifications after the easter holidays,
> > but happy to give some feedback by tomorrow or Thursday as well.
> >
> > Thanks,
> > Timo
> >
> > On 22.04.25 10:40, Jingsong Li wrote:
> > > Thanks Xuannan for driving this discussion.
> > >
> > > At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
> > > etc. are all designing and developing around Variant, and our Flink
> > > support for Variant is very valuable.
> > >
> > > After a rough look at the design, there is no overall problem. It is
> > > designed around Parquet's Variant standard, which is similar to the
> > > overall design of Spark SQL.
> > >
> > > +1 for this.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com>
> > wrote:
> > >>
> > >> Hi devs,
> > >>
> > >> I’d like to start a discussion around FLIP-521: Integrating Variant
> > >> Type into Flink: Enabling Efficient Semi-Structured Data
> > >> Processing[1]. Working with semi-structured data has long been a
> > >> foundational scenario of the Lakehouse. While JSON has traditionally
> > >> served as the primary storage format for such data, its implementation
> > >> as serialized strings introduces significant inefficiencies.
> > >>
> > >> In this FLIP, we integrate the Variant encoding, which is a compact
> > >> binary representation of semi-structured data[2], to improve the
> > >> performance of processing semi-structured data. As Paimon has
> > >> supported the Variant type recently[3], this FLIP would allow Flink to
> > >> further leverage Paimon's storage-layer optimizations, improving
> > >> performance and resource utilization for semi-structured data
> > >> pipelines.
> > >>
> > >> Best,
> > >> Xuannan
> > >>
> > >> [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
> > >> [2]
> > https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
> > >> [3] https://github.com/apache/paimon/issues/4471
> > >
> >
> >
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xuannan Su <su...@gmail.com>.
Hi Timo，
Thanks for the comments!
1. We want the Variant interface to be introduced in the core API, so
we will not use the TableRuntimeException. And a more specific exception
has been added for Variant. Please see the FLIP.
2. I agree that `isScalar()` and `getBytes()` seems more appropriate.
3. This is a good question. After some investigation, I think the
generic is unnecessary.
4. That sounds good to me. I updated the names of the methods.
5. It means timestamp with no time zone, according to the Parquet
variant specification[1], where the type TIMESTAMP has time zone
information.
It is indeed confusing, so I updated the type to stick with the
convention of Flink type system.
6. The missing types are added. And I think we should implement within
the Flink code base for modelling the same binary layout, without
introducing a dependency on Parquet.
7. Package and module information are added.
8. I agree that we can use `JSON_STRING` for the `TO_JSON`. For the
`JSON` method to replace the `PARSE_JSON`, I cannot figure out how it
can work with the current JSON_OBJECT and JSON_ARRAY functions if it
returns a Variant. And we may want to have `PARSE_XML` in the future
that converts XML string to a Variant. Therefore, I am leaning toward
introducing the new `PARSE_JSON` method. What do you think?
9. The SQL/JSON item methods in the SQL standard look good. However, I
think this should be implemented in Calcite, instead of implementing
our own in Flink. And the syntax proposed in the FLIP is actually the
current implementation of the item method for the Variant type[2],
which we just need to put a trivial effort to support after we bump
the Calcite version.
10. After some investigation, I found that just backporting required
classes is not a trivial work. There are many classes to be copied to
the Flink repo. Therefore, instead of going for a non-trivial
temporary solution that will be removed later, we are better off just
bumping the Calcite version, which is inevitable.
11. Yes, the String representation of a variant is a JSON string. We
will put some limitations on casting from/to variant type. In short,
we only allow casting from/to scalar value variant, and an exception
will be thrown in case of type error if `CAST` is used, and null will
be returned if `TRY_CAST` is used. To answer your question,
CAST(variant AS ROW<i INT>) is not valid. And CAST(variant AS INT) is
valid, and it may throw a RuntimeException if the variant type cannot
be cast to an integer. I updated the FLIP to include these
information.
[1] https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
[2] https://calcite.apache.org/docs/reference.html#the-variant-type
On Fri, Apr 25, 2025 at 10:10 PM Timo Walther <tw...@apache.org> wrote:
>
> Hi Xuannan,
>
> sorry for the delay. This is a great addition but needs careful design.
> Here is some feedback that I found while reading the design doc:
>
> 1. Exception design
>
> The FLIP mentions IllegalStateException to be thrown. However, this is a
> very generic exception. We should rather use TableRuntimeException to
> indicate that this is a user's mistake. Or maybe even a dedicated new
> runtime exception type for variants.
>
> 2. Variant.isValue()
>
> The JavaDoc already indicates that a `isScalar()` might be more
> appropriate naming.
>
> 2. Variant.getBinary()
>
> Shall we call this getBytes()? This rather fits to the data type BYTES.
> We are also not calling `getString()` as `getVarchar()`.
>
> 3. VariantBuilder<T>
>
> Could you give some explanation why the VariantBuilder uses a generic?
> Maybe give an example usage in a UDF?
>
> 4. Variant.get() vs. Variant.getField
>
> We should be more explicit in the naming. How about we use
> `Variant.getField(String)` and `Variant.getElement(int)` instead? This
> would also allow us to introduce a `Object Variant.get()` that returns a
> value purely on the `getType` and a `<T> T getAs()` consistent with the
> method `getFieldAs` in the Row class.
>
> 5. Type.TIMESTAMP_NTZ
>
> Do you mean TIMESTAMP_LTZ here?
>
> 6. Implementation of Variant that follow the same layout as the Parquet
> Variant Encoding
>
> Parquet Variant Encoding supports int8, int16, int32. Shouldn't we also
> then support those types in `Variant` class like Short, Byte etc.? And
> especially I noticed that Integer is not supported? We should aim to be
> close to the existing type system.
>
> Also, will we add Parque dependencies or implement this within the Flink
> code base for modelling the same binary layout?
>
> 7. Packages / modules
>
> Please specify where you want to add the classes. Which package/module?
>
> 8. Built-in functions
>
> We recently introduced a `JSON()` function but reserved the use to
> `JSON_OBJECT` and `JSON_ARRAY` only. With variant type support, we can
> allow using this function at any location and return a VARIANT type. I
> guess it maps to `TRY_PARSE_JSON`?
>
> A TO_JSON function is not required as `JSON_STRING` already supports
> arbitrary types and just needs to be extended.
>
> 9. Extract values from a variant
>
> The SQL standard defined SQL/JSON item methods (T865–T878) [1]. It would
> make SQL more readable to support a method per data type.
> E.g. `SELECT integer(variantCol), timestamp(variantCol)`
>
> 10. Bumping Calcite
>
> The FLIP states that you want to bump the version of Apache Calcite to
> 1.39.0. Are you planning to just backport required classes or really
> upgrade to this version? If this is a full upgrade, we should do this in
> stages as history has shown that bumping Calcite is not a
> straight-forward task. But in any case, this would be great to do as we
> could benefit from other features such as lambda support.
>
> 11. String representation and casting rules
>
> Could you elaborate in the FLIP how CAST(variant AS ROW<i INT>) or
> CAST(variant AS INT) behave? Also could you elaborate how the
> `env.executeSql().print()` representation looks like? Will it be JSON?
> Also, Row.toString should support Variant type.
>
> Cheers,
> Timo
>
>
> [1]
> https://peter.eisentraut.org/blog/2023/04/04/sql-2023-is-finished-here-is-whats-new
>
>
> On 25.04.25 13:46, Xuannan Su wrote:
> > Hi everyone,
> >
> > Thank you for all the comments! If there are no further comments, I'd
> > like to close the discussion and start the voting next Monday.
> >
> > Best,
> > Xuannan
> >
> > On Fri, Apr 25, 2025 at 7:41 PM Lincoln Lee <li...@gmail.com> wrote:
> >>
> >> +1 for this FLIP. VARIANT type support will be a great addition to sql.
> >> Look forward to the detailed design of the subsequent shredding
> >> optimizations.
> >>
> >>
> >> Best,
> >> Lincoln Lee
> >>
> >>
> >> Timo Walther <tw...@apache.org> 于2025年4月22日周二 16:51写道：
> >>
> >>> +1 for this feature. Having a VARIANT type makes a lot of sense and
> >>> together with an OBJECT type will make semi-structured data processing
> >>> in Flink easier.
> >>>
> >>> Currently, I'm catching up with notifications after the easter holidays,
> >>> but happy to give some feedback by tomorrow or Thursday as well.
> >>>
> >>> Thanks,
> >>> Timo
> >>>
> >>> On 22.04.25 10:40, Jingsong Li wrote:
> >>>> Thanks Xuannan for driving this discussion.
> >>>>
> >>>> At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
> >>>> etc. are all designing and developing around Variant, and our Flink
> >>>> support for Variant is very valuable.
> >>>>
> >>>> After a rough look at the design, there is no overall problem. It is
> >>>> designed around Parquet's Variant standard, which is similar to the
> >>>> overall design of Spark SQL.
> >>>>
> >>>> +1 for this.
> >>>>
> >>>> Best,
> >>>> Jingsong
> >>>>
> >>>> On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>> Hi devs,
> >>>>>
> >>>>> I’d like to start a discussion around FLIP-521: Integrating Variant
> >>>>> Type into Flink: Enabling Efficient Semi-Structured Data
> >>>>> Processing[1]. Working with semi-structured data has long been a
> >>>>> foundational scenario of the Lakehouse. While JSON has traditionally
> >>>>> served as the primary storage format for such data, its implementation
> >>>>> as serialized strings introduces significant inefficiencies.
> >>>>>
> >>>>> In this FLIP, we integrate the Variant encoding, which is a compact
> >>>>> binary representation of semi-structured data[2], to improve the
> >>>>> performance of processing semi-structured data. As Paimon has
> >>>>> supported the Variant type recently[3], this FLIP would allow Flink to
> >>>>> further leverage Paimon's storage-layer optimizations, improving
> >>>>> performance and resource utilization for semi-structured data
> >>>>> pipelines.
> >>>>>
> >>>>> Best,
> >>>>> Xuannan
> >>>>>
> >>>>> [1]
> >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
> >>>>> [2]
> >>> https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
> >>>>> [3] https://github.com/apache/paimon/issues/4471
> >>>>
> >>>
> >>>
> >
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Timo Walther <tw...@apache.org>.
Hi Xuannan,
sorry for the delay. This is a great addition but needs careful design.
Here is some feedback that I found while reading the design doc:
1. Exception design
The FLIP mentions IllegalStateException to be thrown. However, this is a
very generic exception. We should rather use TableRuntimeException to
indicate that this is a user's mistake. Or maybe even a dedicated new
runtime exception type for variants.
2. Variant.isValue()
The JavaDoc already indicates that a `isScalar()` might be more
appropriate naming.
2. Variant.getBinary()
Shall we call this getBytes()? This rather fits to the data type BYTES.
We are also not calling `getString()` as `getVarchar()`.
3. VariantBuilder<T>
Could you give some explanation why the VariantBuilder uses a generic?
Maybe give an example usage in a UDF?
4. Variant.get() vs. Variant.getField
We should be more explicit in the naming. How about we use
`Variant.getField(String)` and `Variant.getElement(int)` instead? This
would also allow us to introduce a `Object Variant.get()` that returns a
value purely on the `getType` and a `<T> T getAs()` consistent with the
method `getFieldAs` in the Row class.
5. Type.TIMESTAMP_NTZ
Do you mean TIMESTAMP_LTZ here?
6. Implementation of Variant that follow the same layout as the Parquet
Variant Encoding
Parquet Variant Encoding supports int8, int16, int32. Shouldn't we also
then support those types in `Variant` class like Short, Byte etc.? And
especially I noticed that Integer is not supported? We should aim to be
close to the existing type system.
Also, will we add Parque dependencies or implement this within the Flink
code base for modelling the same binary layout?
7. Packages / modules
Please specify where you want to add the classes. Which package/module?
8. Built-in functions
We recently introduced a `JSON()` function but reserved the use to
`JSON_OBJECT` and `JSON_ARRAY` only. With variant type support, we can
allow using this function at any location and return a VARIANT type. I
guess it maps to `TRY_PARSE_JSON`?
A TO_JSON function is not required as `JSON_STRING` already supports
arbitrary types and just needs to be extended.
9. Extract values from a variant
The SQL standard defined SQL/JSON item methods (T865–T878) [1]. It would
make SQL more readable to support a method per data type.
E.g. `SELECT integer(variantCol), timestamp(variantCol)`
10. Bumping Calcite
The FLIP states that you want to bump the version of Apache Calcite to
1.39.0. Are you planning to just backport required classes or really
upgrade to this version? If this is a full upgrade, we should do this in
stages as history has shown that bumping Calcite is not a
straight-forward task. But in any case, this would be great to do as we
could benefit from other features such as lambda support.
11. String representation and casting rules
Could you elaborate in the FLIP how CAST(variant AS ROW<i INT>) or
CAST(variant AS INT) behave? Also could you elaborate how the
`env.executeSql().print()` representation looks like? Will it be JSON?
Also, Row.toString should support Variant type.
Cheers,
Timo
[1]
https://peter.eisentraut.org/blog/2023/04/04/sql-2023-is-finished-here-is-whats-new
On 25.04.25 13:46, Xuannan Su wrote:
> Hi everyone,
>
> Thank you for all the comments! If there are no further comments, I'd
> like to close the discussion and start the voting next Monday.
>
> Best,
> Xuannan
>
> On Fri, Apr 25, 2025 at 7:41 PM Lincoln Lee <li...@gmail.com> wrote:
>>
>> +1 for this FLIP. VARIANT type support will be a great addition to sql.
>> Look forward to the detailed design of the subsequent shredding
>> optimizations.
>>
>>
>> Best,
>> Lincoln Lee
>>
>>
>> Timo Walther <tw...@apache.org> 于2025年4月22日周二 16:51写道：
>>
>>> +1 for this feature. Having a VARIANT type makes a lot of sense and
>>> together with an OBJECT type will make semi-structured data processing
>>> in Flink easier.
>>>
>>> Currently, I'm catching up with notifications after the easter holidays,
>>> but happy to give some feedback by tomorrow or Thursday as well.
>>>
>>> Thanks,
>>> Timo
>>>
>>> On 22.04.25 10:40, Jingsong Li wrote:
>>>> Thanks Xuannan for driving this discussion.
>>>>
>>>> At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
>>>> etc. are all designing and developing around Variant, and our Flink
>>>> support for Variant is very valuable.
>>>>
>>>> After a rough look at the design, there is no overall problem. It is
>>>> designed around Parquet's Variant standard, which is similar to the
>>>> overall design of Spark SQL.
>>>>
>>>> +1 for this.
>>>>
>>>> Best,
>>>> Jingsong
>>>>
>>>> On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com>
>>> wrote:
>>>>>
>>>>> Hi devs,
>>>>>
>>>>> I’d like to start a discussion around FLIP-521: Integrating Variant
>>>>> Type into Flink: Enabling Efficient Semi-Structured Data
>>>>> Processing[1]. Working with semi-structured data has long been a
>>>>> foundational scenario of the Lakehouse. While JSON has traditionally
>>>>> served as the primary storage format for such data, its implementation
>>>>> as serialized strings introduces significant inefficiencies.
>>>>>
>>>>> In this FLIP, we integrate the Variant encoding, which is a compact
>>>>> binary representation of semi-structured data[2], to improve the
>>>>> performance of processing semi-structured data. As Paimon has
>>>>> supported the Variant type recently[3], this FLIP would allow Flink to
>>>>> further leverage Paimon's storage-layer optimizations, improving
>>>>> performance and resource utilization for semi-structured data
>>>>> pipelines.
>>>>>
>>>>> Best,
>>>>> Xuannan
>>>>>
>>>>> [1]
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
>>>>> [2]
>>> https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
>>>>> [3] https://github.com/apache/paimon/issues/4471
>>>>
>>>
>>>
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xuannan Su <su...@gmail.com>.
Hi everyone,
Thank you for all the comments! If there are no further comments, I'd
like to close the discussion and start the voting next Monday.
Best,
Xuannan
On Fri, Apr 25, 2025 at 7:41 PM Lincoln Lee <li...@gmail.com> wrote:
>
> +1 for this FLIP. VARIANT type support will be a great addition to sql.
> Look forward to the detailed design of the subsequent shredding
> optimizations.
>
>
> Best,
> Lincoln Lee
>
>
> Timo Walther <tw...@apache.org> 于2025年4月22日周二 16:51写道：
>
> > +1 for this feature. Having a VARIANT type makes a lot of sense and
> > together with an OBJECT type will make semi-structured data processing
> > in Flink easier.
> >
> > Currently, I'm catching up with notifications after the easter holidays,
> > but happy to give some feedback by tomorrow or Thursday as well.
> >
> > Thanks,
> > Timo
> >
> > On 22.04.25 10:40, Jingsong Li wrote:
> > > Thanks Xuannan for driving this discussion.
> > >
> > > At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
> > > etc. are all designing and developing around Variant, and our Flink
> > > support for Variant is very valuable.
> > >
> > > After a rough look at the design, there is no overall problem. It is
> > > designed around Parquet's Variant standard, which is similar to the
> > > overall design of Spark SQL.
> > >
> > > +1 for this.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com>
> > wrote:
> > >>
> > >> Hi devs,
> > >>
> > >> I’d like to start a discussion around FLIP-521: Integrating Variant
> > >> Type into Flink: Enabling Efficient Semi-Structured Data
> > >> Processing[1]. Working with semi-structured data has long been a
> > >> foundational scenario of the Lakehouse. While JSON has traditionally
> > >> served as the primary storage format for such data, its implementation
> > >> as serialized strings introduces significant inefficiencies.
> > >>
> > >> In this FLIP, we integrate the Variant encoding, which is a compact
> > >> binary representation of semi-structured data[2], to improve the
> > >> performance of processing semi-structured data. As Paimon has
> > >> supported the Variant type recently[3], this FLIP would allow Flink to
> > >> further leverage Paimon's storage-layer optimizations, improving
> > >> performance and resource utilization for semi-structured data
> > >> pipelines.
> > >>
> > >> Best,
> > >> Xuannan
> > >>
> > >> [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
> > >> [2]
> > https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
> > >> [3] https://github.com/apache/paimon/issues/4471
> > >
> >
> >
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Lincoln Lee <li...@gmail.com>.
+1 for this FLIP. VARIANT type support will be a great addition to sql.
Look forward to the detailed design of the subsequent shredding
optimizations.
Best,
Lincoln Lee
Timo Walther <tw...@apache.org> 于2025年4月22日周二 16:51写道：
> +1 for this feature. Having a VARIANT type makes a lot of sense and
> together with an OBJECT type will make semi-structured data processing
> in Flink easier.
>
> Currently, I'm catching up with notifications after the easter holidays,
> but happy to give some feedback by tomorrow or Thursday as well.
>
> Thanks,
> Timo
>
> On 22.04.25 10:40, Jingsong Li wrote:
> > Thanks Xuannan for driving this discussion.
> >
> > At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
> > etc. are all designing and developing around Variant, and our Flink
> > support for Variant is very valuable.
> >
> > After a rough look at the design, there is no overall problem. It is
> > designed around Parquet's Variant standard, which is similar to the
> > overall design of Spark SQL.
> >
> > +1 for this.
> >
> > Best,
> > Jingsong
> >
> > On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com>
> wrote:
> >>
> >> Hi devs,
> >>
> >> I’d like to start a discussion around FLIP-521: Integrating Variant
> >> Type into Flink: Enabling Efficient Semi-Structured Data
> >> Processing[1]. Working with semi-structured data has long been a
> >> foundational scenario of the Lakehouse. While JSON has traditionally
> >> served as the primary storage format for such data, its implementation
> >> as serialized strings introduces significant inefficiencies.
> >>
> >> In this FLIP, we integrate the Variant encoding, which is a compact
> >> binary representation of semi-structured data[2], to improve the
> >> performance of processing semi-structured data. As Paimon has
> >> supported the Variant type recently[3], this FLIP would allow Flink to
> >> further leverage Paimon's storage-layer optimizations, improving
> >> performance and resource utilization for semi-structured data
> >> pipelines.
> >>
> >> Best,
> >> Xuannan
> >>
> >> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
> >> [2]
> https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
> >> [3] https://github.com/apache/paimon/issues/4471
> >
>
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Timo Walther <tw...@apache.org>.
+1 for this feature. Having a VARIANT type makes a lot of sense and
together with an OBJECT type will make semi-structured data processing
in Flink easier.
Currently, I'm catching up with notifications after the easter holidays,
but happy to give some feedback by tomorrow or Thursday as well.
Thanks,
Timo
On 22.04.25 10:40, Jingsong Li wrote:
> Thanks Xuannan for driving this discussion.
>
> At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
> etc. are all designing and developing around Variant, and our Flink
> support for Variant is very valuable.
>
> After a rough look at the design, there is no overall problem. It is
> designed around Parquet's Variant standard, which is similar to the
> overall design of Spark SQL.
>
> +1 for this.
>
> Best,
> Jingsong
>
> On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com> wrote:
>>
>> Hi devs,
>>
>> I’d like to start a discussion around FLIP-521: Integrating Variant
>> Type into Flink: Enabling Efficient Semi-Structured Data
>> Processing[1]. Working with semi-structured data has long been a
>> foundational scenario of the Lakehouse. While JSON has traditionally
>> served as the primary storage format for such data, its implementation
>> as serialized strings introduces significant inefficiencies.
>>
>> In this FLIP, we integrate the Variant encoding, which is a compact
>> binary representation of semi-structured data[2], to improve the
>> performance of processing semi-structured data. As Paimon has
>> supported the Variant type recently[3], this FLIP would allow Flink to
>> further leverage Paimon's storage-layer optimizations, improving
>> performance and resource utilization for semi-structured data
>> pipelines.
>>
>> Best,
>> Xuannan
>>
>> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
>> [2] https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
>> [3] https://github.com/apache/paimon/issues/4471
>
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Jingsong Li <ji...@gmail.com>.
Thanks Xuannan for driving this discussion.
At present, communities such as Apache Iceberg, Delta, Spark, Parquet,
etc. are all designing and developing around Variant, and our Flink
support for Variant is very valuable.
After a rough look at the design, there is no overall problem. It is
designed around Parquet's Variant standard, which is similar to the
overall design of Spark SQL.
+1 for this.
Best,
Jingsong
On Mon, Apr 14, 2025 at 6:12 PM Xuannan Su <su...@gmail.com> wrote:
>
> Hi devs,
>
> I’d like to start a discussion around FLIP-521: Integrating Variant
> Type into Flink: Enabling Efficient Semi-Structured Data
> Processing[1]. Working with semi-structured data has long been a
> foundational scenario of the Lakehouse. While JSON has traditionally
> served as the primary storage format for such data, its implementation
> as serialized strings introduces significant inefficiencies.
>
> In this FLIP, we integrate the Variant encoding, which is a compact
> binary representation of semi-structured data[2], to improve the
> performance of processing semi-structured data. As Paimon has
> supported the Variant type recently[3], this FLIP would allow Flink to
> further leverage Paimon's storage-layer optimizations, improving
> performance and resource utilization for semi-structured data
> pipelines.
>
> Best,
> Xuannan
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
> [2] https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
> [3] https://github.com/apache/paimon/issues/4471
Re: [DISCUSS] FLIP-521: Integrating Variant Type into Flink: Enabling Efficient Semi-Structured Data Processing
Posted by Xintong Song <to...@gmail.com>.
+1 for this proposal.
Storing all structured, semi-structured and unstructured data at one place
has been one of the main design goals since day 1 of data lake.
Semi-structured / unstructured data processing is becoming even more
crucial in the AI era. As already supported by lake formats like Iceberg
and Paimon and processing engines like Spark and StarRocks, I think Flink
should also have that support, in order to complete the semi-structured
data processing picture in the lakehouse architecture.
The FLIP looks good to me in general.
Best,
Xintong
On Mon, Apr 14, 2025 at 6:13 PM Xuannan Su <su...@gmail.com> wrote:
> Hi devs,
>
> I’d like to start a discussion around FLIP-521: Integrating Variant
> Type into Flink: Enabling Efficient Semi-Structured Data
> Processing[1]. Working with semi-structured data has long been a
> foundational scenario of the Lakehouse. While JSON has traditionally
> served as the primary storage format for such data, its implementation
> as serialized strings introduces significant inefficiencies.
>
> In this FLIP, we integrate the Variant encoding, which is a compact
> binary representation of semi-structured data[2], to improve the
> performance of processing semi-structured data. As Paimon has
> supported the Variant type recently[3], this FLIP would allow Flink to
> further leverage Paimon's storage-layer optimizations, improving
> performance and resource utilization for semi-structured data
> pipelines.
>
> Best,
> Xuannan
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-521%3A+Integrating+Variant+Type+into+Flink%3A+Enabling+Efficient+Semi-Structured+Data+Processing
> [2]
> https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
> [3] https://github.com/apache/paimon/issues/4471
>