You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Moritz Manner <mo...@gmail.com> on 2026/09/11 12:54:59 UTC
[DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Hey everyone,
I'd like to start a discussion on FLIP-612: Native XML Functions for Flink
SQL [1].
The goal is native XML parsing in Flink SQL, parsing XML into a VARIANT
instead of requiring custom UDFs.
The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML turn an XML
string into a VARIANT, and XML_STRING turns a VARIANT back into an XML
string.
This mirrors the existing JSON functions PARSE_JSON, TRY_PARSE_JSON, and
JSON_STRING.
Once XML is a VARIANT, you query it the same way as the output of PARSE_JSON:
with the variant accessors (variant.key, variant['key'], variant[index])
and a CAST to the type you need.
The main part is the XML→VARIANT mapping: attributes and text go into @/$
fields, and since Variant objects store fields sorted by key rather than in
document order, a # field is used to recover the original order between
differently-named siblings. Details and examples are in the FLIP.
It's a small, additive API, no new SQL grammar or Table API methods needed.
Looking forward to your thoughts!
Best,
Moritz
[1]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612+Native+XML+Functions+for+Flink+SQL
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Timo Walther <tw...@apache.org>.
Thank you for updating the FLIP. Feel free to start the VOTE now.
Remaining minor changes we can still discuss and address during PR
review. The overall design LGTM.
Thanks,
Timo
On 24.09.26 14:30, Moritz Manner wrote:
> Hi all,
> I updated the FLIP with the points from the recent discussion. The main
> changes:
>
> - force_array option: PARSE_XML and TRY_PARSE_XML now take an optional
> boolean (default false). When true, every element maps to an array even if
> it appears once, so the shape stays stable. The compact format is still the
> default.
> - # field: dropped when there is only a single distinct key. A single
> repeated tag already keeps its order in its array, so # carries no
> information there.
> - xsi / namespaces: added the round-trip behavior and spelled out the
> literal-prefix limitation in both directions (thanks Sai for pointing out
> the second one).
>
> On namespaces: we match xsi:nil and xsi:type by the literal xsi: prefix,
> not by the resolved namespace. In real documents these attributes might
> appear without a proper xmlns:xsi declaration, and literal matching still
> handles them. A namespace-aware parser would need the prefix declared and
> would reject those.
> The tradeoff is that we rely on the xsi convention. If you bind xsi to a
> non-standard namespace, or use a different prefix for the schema-instance
> namespace, it won't be handled correctly. However, that's unusual in
> practice.
>
> Please have another look and let me know if anything is still unclear. If
> there are no more concerns, we can start a vote soon.
>
> Best,
> Moritz
>
> On Tue, 22 Sept 2026 at 15:25, Timo Walther <tw...@apache.org> wrote:
>
>> Hi Moritz,
>>
>> thanks for considering my feedback. +1 for a compact representation by
>> default but an additional parameter that enables consistent structure
>> across multiple XML document.
>>
>> +1 for voting from my side, modulo the remaining comment from Sai.
>>
>> Thanks,
>> Timo
>>
>>
>> On 22.09.26 07:22, Krishna Sai wrote:
>>> Hi Moritz,
>>>
>>> That answers it, thanks. Since PARSE_XML consumes xmlns:xsi and
>> XML_STRING
>>> is the only thing that emits it, the duplicate can't happen.
>>>
>>> On the note you're adding, the other direction might be worth a line too.
>>> The FLIP already records a different prefix bound to the XSD-instance
>>> namespace. The opposite case is xsi bound to something else:
>>>
>>> <root xmlns:xsi="urn:my-app"><a xsi:nil="true"/></root>
>>>
>>> A namespace-aware parser reads that attribute as {urn:my-app}nil, an
>>> ordinary application attribute. PARSE_XML would take it as a null,
>> consume
>>> the declaration, and XML_STRING would emit the XSD-instance one, so the
>>> prefix comes back bound to a different namespace than it went in. Unlike
>>> the unused-declaration case, that one changes meaning rather than bytes.
>>>
>>> Best,
>>> Sai Krishna Sepuri
>>
>>
>
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Moritz Manner <mo...@gmail.com>.
Hi all,
I updated the FLIP with the points from the recent discussion. The main
changes:
- force_array option: PARSE_XML and TRY_PARSE_XML now take an optional
boolean (default false). When true, every element maps to an array even if
it appears once, so the shape stays stable. The compact format is still the
default.
- # field: dropped when there is only a single distinct key. A single
repeated tag already keeps its order in its array, so # carries no
information there.
- xsi / namespaces: added the round-trip behavior and spelled out the
literal-prefix limitation in both directions (thanks Sai for pointing out
the second one).
On namespaces: we match xsi:nil and xsi:type by the literal xsi: prefix,
not by the resolved namespace. In real documents these attributes might
appear without a proper xmlns:xsi declaration, and literal matching still
handles them. A namespace-aware parser would need the prefix declared and
would reject those.
The tradeoff is that we rely on the xsi convention. If you bind xsi to a
non-standard namespace, or use a different prefix for the schema-instance
namespace, it won't be handled correctly. However, that's unusual in
practice.
Please have another look and let me know if anything is still unclear. If
there are no more concerns, we can start a vote soon.
Best,
Moritz
On Tue, 22 Sept 2026 at 15:25, Timo Walther <tw...@apache.org> wrote:
> Hi Moritz,
>
> thanks for considering my feedback. +1 for a compact representation by
> default but an additional parameter that enables consistent structure
> across multiple XML document.
>
> +1 for voting from my side, modulo the remaining comment from Sai.
>
> Thanks,
> Timo
>
>
> On 22.09.26 07:22, Krishna Sai wrote:
> > Hi Moritz,
> >
> > That answers it, thanks. Since PARSE_XML consumes xmlns:xsi and
> XML_STRING
> > is the only thing that emits it, the duplicate can't happen.
> >
> > On the note you're adding, the other direction might be worth a line too.
> > The FLIP already records a different prefix bound to the XSD-instance
> > namespace. The opposite case is xsi bound to something else:
> >
> > <root xmlns:xsi="urn:my-app"><a xsi:nil="true"/></root>
> >
> > A namespace-aware parser reads that attribute as {urn:my-app}nil, an
> > ordinary application attribute. PARSE_XML would take it as a null,
> consume
> > the declaration, and XML_STRING would emit the XSD-instance one, so the
> > prefix comes back bound to a different namespace than it went in. Unlike
> > the unused-declaration case, that one changes meaning rather than bytes.
> >
> > Best,
> > Sai Krishna Sepuri
>
>
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Timo Walther <tw...@apache.org>.
Hi Moritz,
thanks for considering my feedback. +1 for a compact representation by
default but an additional parameter that enables consistent structure
across multiple XML document.
+1 for voting from my side, modulo the remaining comment from Sai.
Thanks,
Timo
On 22.09.26 07:22, Krishna Sai wrote:
> Hi Moritz,
>
> That answers it, thanks. Since PARSE_XML consumes xmlns:xsi and XML_STRING
> is the only thing that emits it, the duplicate can't happen.
>
> On the note you're adding, the other direction might be worth a line too.
> The FLIP already records a different prefix bound to the XSD-instance
> namespace. The opposite case is xsi bound to something else:
>
> <root xmlns:xsi="urn:my-app"><a xsi:nil="true"/></root>
>
> A namespace-aware parser reads that attribute as {urn:my-app}nil, an
> ordinary application attribute. PARSE_XML would take it as a null, consume
> the declaration, and XML_STRING would emit the XSD-instance one, so the
> prefix comes back bound to a different namespace than it went in. Unlike
> the unused-declaration case, that one changes meaning rather than bytes.
>
> Best,
> Sai Krishna Sepuri
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Krishna Sai <sa...@gmail.com>.
Hi Moritz,
That answers it, thanks. Since PARSE_XML consumes xmlns:xsi and XML_STRING
is the only thing that emits it, the duplicate can't happen.
On the note you're adding, the other direction might be worth a line too.
The FLIP already records a different prefix bound to the XSD-instance
namespace. The opposite case is xsi bound to something else:
<root xmlns:xsi="urn:my-app"><a xsi:nil="true"/></root>
A namespace-aware parser reads that attribute as {urn:my-app}nil, an
ordinary application attribute. PARSE_XML would take it as a null, consume
the declaration, and XML_STRING would emit the XSD-instance one, so the
prefix comes back bound to a different namespace than it went in. Unlike
the unused-declaration case, that one changes meaning rather than bytes.
Best,
Sai Krishna Sepuri
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Moritz Manner <mo...@gmail.com>.
Hi Sai, hi Timo,
@Sai:
Good point! As you mentioned PARSE_XML isn't namespace-aware and keeps
namespaces verbatim: xmlns:foo becomes a normal @xmlns:foo attribute.
xmlns:xsi is the exception. Since PARSE_XML interprets xsi:nil and
xsi:type, it consumes the xmlns:xsi declaration instead of keeping it. It's
then the job of XML_STRING to decide if it's needed again on the output: it
declares xmlns:xsi on the root whenever the output has any xsi: attribute,
and nothing otherwise. That way the declaration can't land on the root
twice. PARSE_XML consumes the input's xmlns:xsi, and XML_STRING is the only
place that emits it. Your example round-trips to the same well-formed XML.
The one tradeoff: an xmlns:xsi that's declared but never used gets dropped
(semantically equivalent, just not byte-identical). I'll add a note to the
FLIP. Does that make sense and answer your question?
@Timo:
The a.b.c ergonomics are a good point, indexing everywhere is a real
annoyance for simple XMLs, which is the common case. I think an opt-in flag
can cover both cases: we keep compact as the default so a.b.c works out of
the box, and add an optional boolean to PARSE_XML (default false) that
wraps every element in an array, even when it only appears once. That also
covers Fabian's point: for a batch of documents with the same schema where
you want a stable structure regardless of occurrence count, you turn it on
and always get arrays.
Best,
Moritz
On Fri, 18 Sept 2026 at 14:36, Timo Walther <tw...@apache.org> wrote:
> Hi all,
>
> I'm wondering if defaults shouldn't be the other way around? First, we
> should aim for an `a.b.c` access syntax as much as possible. An array
> syntax `a[1].b[1].c` is very cumbersome. We can also add a XML_PATH
> function in the future that uses a string path e.g. XML_PATH("a.b.c")
> and works dynamically with either fields or arrays under the same
> parent. JSON_VALUE has a similar logic if I remember correctly.
>
> Alternatively, I could enable `a[1].b[1].c` with an additional parameter
> once we collected enough feedback.
>
> Cheers,
> Timo
>
>
> On 18.09.26 10:19, Fabian Hüske via dev wrote:
> > Hi Moritz, hi all,
> >
> >> The idea would be to make ARRAY the default for all elements, so the
> >> structure stays the same no matter how many elements are actually there.
> >> The downside is that for simple XMLs, where most elements only ever
> appear
> >> once, you'd need to add [1] everywhere just to access them.
> >
> >> One option would be to start with ARRAY by default and see how it goes
> >> based on feedback. If it turns out to be too cumbersome for simple XMLs,
> > we
> >> could add an opt-out parameter later, that would be a
> backward-compatible
> >> addition so it doesn't need to be solved now.
> >> Does this make sense, or is there a better way to approach this?
> >
> > Sounds good to me, thanks!
> >
> > Best, Fabian
> >
> > On Thu, Sep 17, 2026 at 5:30 PM Krishna Sai <sa...@gmail.com> wrote:
> >
> >> Hi Moritz,
> >>
> >> Nice FLIP. The # field is a neat solution, and I checked on master that
> >> Variant does sort object keys, so it is doing necessary work.
> >>
> >> One small question: what does PARSE_XML do with an xmlns declaration in
> >> the input? The parser is not namespace-aware, so xmlns:xsi is an
> >> attribute, and by the @name rule it would land as "@xmlns:xsi". If
> >> XML_STRING writes that back and also adds its own xmlns:xsi declaration
> >> for xsi:nil, the root would carry that attribute twice, which is not
> >> well-formed, so that Variant would not round trip.
> >>
> >> <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
> >> <a xsi:nil="true"/>
> >> </root>
> >>
> >> Dropping xmlns at parse time would instead leave other prefixes
> >> undeclared. Maybe I am misreading the mapping, but a line either way
> >> might help.
> >>
> >> Best,
> >> Sai Krishna Sepuri
> >>
> >> On Thu, Sep 17, 2026 at 7:51 PM Moritz Manner <mo...@gmail.com>
> >> wrote:
> >>>
> >>> Hi Fabian,
> >>>
> >>> Good point, and I agree this is a real problem. A consistent VARIANT
> >>> structure across documents with the same schema is important,
> especially
> >>> for streaming SQL where you don't want CASE WHEN everywhere, or a job
> >>> failing because it tries to access an array that doesn't exist.
> >> Initially,
> >>> the structure optimized for queryability, i.e. a single element stays a
> >>> single node instead of an array. But I think consistency should
> probably
> >>> win here.
> >>>
> >>> The idea would be to make ARRAY the default for all elements, so the
> >>> structure stays the same no matter how many elements are actually
> there.
> >>> The downside is that for simple XMLs, where most elements only ever
> >> appear
> >>> once, you'd need to add [1] everywhere just to access them.
> >>>
> >>> One option would be to start with ARRAY by default and see how it goes
> >>> based on feedback. If it turns out to be too cumbersome for simple
> XMLs,
> >> we
> >>> could add an opt-out parameter later, that would be a
> backward-compatible
> >>> addition so it doesn't need to be solved now.
> >>> Does this make sense, or is there a better way to approach this?
> >>>
> >>> Best,
> >>> Moritz
> >>>
> >>> On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev <
> >> dev@flink.apache.org>
> >>> wrote:
> >>>
> >>>> Hi everyone and thanks for the proposal Moritz,
> >>>>
> >>>> The FLIP reads very well and I agree (mostly) with the design.
> >>>>
> >>>> I have only one concern, which is about the representation of repeated
> >>>> elements as arrays.
> >>>> I think it is very important that a batch of XML documents with the
> >> same
> >>>> structure converts into an identical VARIANT structure.
> >>>> If I understand the current proposal correctly, this would not be the
> >> case
> >>>> for two documents: one with a single element and one with a repeated
> >>>> element.
> >>>> The first document would have a single node, while the second document
> >>>> would have an array.
> >>>> If you want to access the nodes, you would need a cumbersome CASE WHEN
> >>>> construct to handle both cases.
> >>>> It would be better to convert both documents into a VARIANT containing
> >> an
> >>>> ARRAY (one only holding a single element).
> >>>>
> >>>> Since deriving the right conversion strategy from a single XML
> document
> >>>> isn't possible, could we add an optional argument to pass conversion
> >> hints?
> >>>> There might be other solutions to this problem.
> >>>>
> >>>> IMO, it would also be fine to continue with this FLIP and address this
> >>>> issue later, but having a proposal for it would be good.
> >>>> Do you have any thoughts on this, Moritz?
> >>>>
> >>>> Best, Fabian
> >>>>
> >>>>
> >>>> On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <tw...@apache.org>
> >> wrote:
> >>>>
> >>>>> Hi Moritz,
> >>>>>
> >>>>> this is an excellent design document with the right level of detail.
> >> I
> >>>>> really like the design, it is a nice mixture of concerns. The result
> >> of
> >>>>> PARSE_XML looks and feels like a VARIANT while the XML content is
> >>>>> preserved in a lossless fashion and all attributes and text is
> >>>>> accessible. It is a nice combination of what other vendors offer
> >> (modulo
> >>>>> the historical legacy they have to deal with).
> >>>>>
> >>>>> Looking at various UDF implementations, XML handling is by far the
> >> most
> >>>>> important reason for the need of a custom function. So +1 for this
> >> FLIP.
> >>>>>
> >>>>> Thanks,
> >>>>> Timo
> >>>>>
> >>>>> On 11.09.26 14:54, Moritz Manner wrote:
> >>>>>> Hey everyone,
> >>>>>>
> >>>>>> I'd like to start a discussion on FLIP-612: Native XML Functions
> >> for
> >>>>> Flink
> >>>>>> SQL [1].
> >>>>>>
> >>>>>> The goal is native XML parsing in Flink SQL, parsing XML into a
> >> VARIANT
> >>>>>> instead of requiring custom UDFs.
> >>>>>>
> >>>>>> The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML
> >> turn an
> >>>>> XML
> >>>>>> string into a VARIANT, and XML_STRING turns a VARIANT back into an
> >> XML
> >>>>>> string.
> >>>>>> This mirrors the existing JSON functions PARSE_JSON,
> >> TRY_PARSE_JSON,
> >>>> and
> >>>>>> JSON_STRING.
> >>>>>>
> >>>>>> Once XML is a VARIANT, you query it the same way as the output of
> >>>>> PARSE_JSON:
> >>>>>> with the variant accessors (variant.key, variant['key'],
> >>>> variant[index])
> >>>>>> and a CAST to the type you need.
> >>>>>>
> >>>>>> The main part is the XML→VARIANT mapping: attributes and text go
> >> into
> >>>> @/$
> >>>>>> fields, and since Variant objects store fields sorted by key rather
> >>>> than
> >>>>> in
> >>>>>> document order, a # field is used to recover the original order
> >> between
> >>>>>> differently-named siblings. Details and examples are in the FLIP.
> >>>>>>
> >>>>>> It's a small, additive API, no new SQL grammar or Table API methods
> >>>>> needed.
> >>>>>>
> >>>>>> Looking forward to your thoughts!
> >>>>>>
> >>>>>> Best,
> >>>>>> Moritz
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>
> >>>>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$
> >>>>>
> >>>>>
> >>>>
> >>
> >
>
>
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Timo Walther <tw...@apache.org>.
Hi all,
I'm wondering if defaults shouldn't be the other way around? First, we
should aim for an `a.b.c` access syntax as much as possible. An array
syntax `a[1].b[1].c` is very cumbersome. We can also add a XML_PATH
function in the future that uses a string path e.g. XML_PATH("a.b.c")
and works dynamically with either fields or arrays under the same
parent. JSON_VALUE has a similar logic if I remember correctly.
Alternatively, I could enable `a[1].b[1].c` with an additional parameter
once we collected enough feedback.
Cheers,
Timo
On 18.09.26 10:19, Fabian Hüske via dev wrote:
> Hi Moritz, hi all,
>
>> The idea would be to make ARRAY the default for all elements, so the
>> structure stays the same no matter how many elements are actually there.
>> The downside is that for simple XMLs, where most elements only ever appear
>> once, you'd need to add [1] everywhere just to access them.
>
>> One option would be to start with ARRAY by default and see how it goes
>> based on feedback. If it turns out to be too cumbersome for simple XMLs,
> we
>> could add an opt-out parameter later, that would be a backward-compatible
>> addition so it doesn't need to be solved now.
>> Does this make sense, or is there a better way to approach this?
>
> Sounds good to me, thanks!
>
> Best, Fabian
>
> On Thu, Sep 17, 2026 at 5:30 PM Krishna Sai <sa...@gmail.com> wrote:
>
>> Hi Moritz,
>>
>> Nice FLIP. The # field is a neat solution, and I checked on master that
>> Variant does sort object keys, so it is doing necessary work.
>>
>> One small question: what does PARSE_XML do with an xmlns declaration in
>> the input? The parser is not namespace-aware, so xmlns:xsi is an
>> attribute, and by the @name rule it would land as "@xmlns:xsi". If
>> XML_STRING writes that back and also adds its own xmlns:xsi declaration
>> for xsi:nil, the root would carry that attribute twice, which is not
>> well-formed, so that Variant would not round trip.
>>
>> <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
>> <a xsi:nil="true"/>
>> </root>
>>
>> Dropping xmlns at parse time would instead leave other prefixes
>> undeclared. Maybe I am misreading the mapping, but a line either way
>> might help.
>>
>> Best,
>> Sai Krishna Sepuri
>>
>> On Thu, Sep 17, 2026 at 7:51 PM Moritz Manner <mo...@gmail.com>
>> wrote:
>>>
>>> Hi Fabian,
>>>
>>> Good point, and I agree this is a real problem. A consistent VARIANT
>>> structure across documents with the same schema is important, especially
>>> for streaming SQL where you don't want CASE WHEN everywhere, or a job
>>> failing because it tries to access an array that doesn't exist.
>> Initially,
>>> the structure optimized for queryability, i.e. a single element stays a
>>> single node instead of an array. But I think consistency should probably
>>> win here.
>>>
>>> The idea would be to make ARRAY the default for all elements, so the
>>> structure stays the same no matter how many elements are actually there.
>>> The downside is that for simple XMLs, where most elements only ever
>> appear
>>> once, you'd need to add [1] everywhere just to access them.
>>>
>>> One option would be to start with ARRAY by default and see how it goes
>>> based on feedback. If it turns out to be too cumbersome for simple XMLs,
>> we
>>> could add an opt-out parameter later, that would be a backward-compatible
>>> addition so it doesn't need to be solved now.
>>> Does this make sense, or is there a better way to approach this?
>>>
>>> Best,
>>> Moritz
>>>
>>> On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev <
>> dev@flink.apache.org>
>>> wrote:
>>>
>>>> Hi everyone and thanks for the proposal Moritz,
>>>>
>>>> The FLIP reads very well and I agree (mostly) with the design.
>>>>
>>>> I have only one concern, which is about the representation of repeated
>>>> elements as arrays.
>>>> I think it is very important that a batch of XML documents with the
>> same
>>>> structure converts into an identical VARIANT structure.
>>>> If I understand the current proposal correctly, this would not be the
>> case
>>>> for two documents: one with a single element and one with a repeated
>>>> element.
>>>> The first document would have a single node, while the second document
>>>> would have an array.
>>>> If you want to access the nodes, you would need a cumbersome CASE WHEN
>>>> construct to handle both cases.
>>>> It would be better to convert both documents into a VARIANT containing
>> an
>>>> ARRAY (one only holding a single element).
>>>>
>>>> Since deriving the right conversion strategy from a single XML document
>>>> isn't possible, could we add an optional argument to pass conversion
>> hints?
>>>> There might be other solutions to this problem.
>>>>
>>>> IMO, it would also be fine to continue with this FLIP and address this
>>>> issue later, but having a proposal for it would be good.
>>>> Do you have any thoughts on this, Moritz?
>>>>
>>>> Best, Fabian
>>>>
>>>>
>>>> On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <tw...@apache.org>
>> wrote:
>>>>
>>>>> Hi Moritz,
>>>>>
>>>>> this is an excellent design document with the right level of detail.
>> I
>>>>> really like the design, it is a nice mixture of concerns. The result
>> of
>>>>> PARSE_XML looks and feels like a VARIANT while the XML content is
>>>>> preserved in a lossless fashion and all attributes and text is
>>>>> accessible. It is a nice combination of what other vendors offer
>> (modulo
>>>>> the historical legacy they have to deal with).
>>>>>
>>>>> Looking at various UDF implementations, XML handling is by far the
>> most
>>>>> important reason for the need of a custom function. So +1 for this
>> FLIP.
>>>>>
>>>>> Thanks,
>>>>> Timo
>>>>>
>>>>> On 11.09.26 14:54, Moritz Manner wrote:
>>>>>> Hey everyone,
>>>>>>
>>>>>> I'd like to start a discussion on FLIP-612: Native XML Functions
>> for
>>>>> Flink
>>>>>> SQL [1].
>>>>>>
>>>>>> The goal is native XML parsing in Flink SQL, parsing XML into a
>> VARIANT
>>>>>> instead of requiring custom UDFs.
>>>>>>
>>>>>> The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML
>> turn an
>>>>> XML
>>>>>> string into a VARIANT, and XML_STRING turns a VARIANT back into an
>> XML
>>>>>> string.
>>>>>> This mirrors the existing JSON functions PARSE_JSON,
>> TRY_PARSE_JSON,
>>>> and
>>>>>> JSON_STRING.
>>>>>>
>>>>>> Once XML is a VARIANT, you query it the same way as the output of
>>>>> PARSE_JSON:
>>>>>> with the variant accessors (variant.key, variant['key'],
>>>> variant[index])
>>>>>> and a CAST to the type you need.
>>>>>>
>>>>>> The main part is the XML→VARIANT mapping: attributes and text go
>> into
>>>> @/$
>>>>>> fields, and since Variant objects store fields sorted by key rather
>>>> than
>>>>> in
>>>>>> document order, a # field is used to recover the original order
>> between
>>>>>> differently-named siblings. Details and examples are in the FLIP.
>>>>>>
>>>>>> It's a small, additive API, no new SQL grammar or Table API methods
>>>>> needed.
>>>>>>
>>>>>> Looking forward to your thoughts!
>>>>>>
>>>>>> Best,
>>>>>> Moritz
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>
>>>>
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$
>>>>>
>>>>>
>>>>
>>
>
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Fabian Hüske via dev <de...@flink.apache.org>.
Hi Moritz, hi all,
> The idea would be to make ARRAY the default for all elements, so the
> structure stays the same no matter how many elements are actually there.
> The downside is that for simple XMLs, where most elements only ever appear
> once, you'd need to add [1] everywhere just to access them.
> One option would be to start with ARRAY by default and see how it goes
> based on feedback. If it turns out to be too cumbersome for simple XMLs,
we
> could add an opt-out parameter later, that would be a backward-compatible
> addition so it doesn't need to be solved now.
> Does this make sense, or is there a better way to approach this?
Sounds good to me, thanks!
Best, Fabian
On Thu, Sep 17, 2026 at 5:30 PM Krishna Sai <sa...@gmail.com> wrote:
> Hi Moritz,
>
> Nice FLIP. The # field is a neat solution, and I checked on master that
> Variant does sort object keys, so it is doing necessary work.
>
> One small question: what does PARSE_XML do with an xmlns declaration in
> the input? The parser is not namespace-aware, so xmlns:xsi is an
> attribute, and by the @name rule it would land as "@xmlns:xsi". If
> XML_STRING writes that back and also adds its own xmlns:xsi declaration
> for xsi:nil, the root would carry that attribute twice, which is not
> well-formed, so that Variant would not round trip.
>
> <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
> <a xsi:nil="true"/>
> </root>
>
> Dropping xmlns at parse time would instead leave other prefixes
> undeclared. Maybe I am misreading the mapping, but a line either way
> might help.
>
> Best,
> Sai Krishna Sepuri
>
> On Thu, Sep 17, 2026 at 7:51 PM Moritz Manner <mo...@gmail.com>
> wrote:
> >
> > Hi Fabian,
> >
> > Good point, and I agree this is a real problem. A consistent VARIANT
> > structure across documents with the same schema is important, especially
> > for streaming SQL where you don't want CASE WHEN everywhere, or a job
> > failing because it tries to access an array that doesn't exist.
> Initially,
> > the structure optimized for queryability, i.e. a single element stays a
> > single node instead of an array. But I think consistency should probably
> > win here.
> >
> > The idea would be to make ARRAY the default for all elements, so the
> > structure stays the same no matter how many elements are actually there.
> > The downside is that for simple XMLs, where most elements only ever
> appear
> > once, you'd need to add [1] everywhere just to access them.
> >
> > One option would be to start with ARRAY by default and see how it goes
> > based on feedback. If it turns out to be too cumbersome for simple XMLs,
> we
> > could add an opt-out parameter later, that would be a backward-compatible
> > addition so it doesn't need to be solved now.
> > Does this make sense, or is there a better way to approach this?
> >
> > Best,
> > Moritz
> >
> > On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev <
> dev@flink.apache.org>
> > wrote:
> >
> > > Hi everyone and thanks for the proposal Moritz,
> > >
> > > The FLIP reads very well and I agree (mostly) with the design.
> > >
> > > I have only one concern, which is about the representation of repeated
> > > elements as arrays.
> > > I think it is very important that a batch of XML documents with the
> same
> > > structure converts into an identical VARIANT structure.
> > > If I understand the current proposal correctly, this would not be the
> case
> > > for two documents: one with a single element and one with a repeated
> > > element.
> > > The first document would have a single node, while the second document
> > > would have an array.
> > > If you want to access the nodes, you would need a cumbersome CASE WHEN
> > > construct to handle both cases.
> > > It would be better to convert both documents into a VARIANT containing
> an
> > > ARRAY (one only holding a single element).
> > >
> > > Since deriving the right conversion strategy from a single XML document
> > > isn't possible, could we add an optional argument to pass conversion
> hints?
> > > There might be other solutions to this problem.
> > >
> > > IMO, it would also be fine to continue with this FLIP and address this
> > > issue later, but having a proposal for it would be good.
> > > Do you have any thoughts on this, Moritz?
> > >
> > > Best, Fabian
> > >
> > >
> > > On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <tw...@apache.org>
> wrote:
> > >
> > > > Hi Moritz,
> > > >
> > > > this is an excellent design document with the right level of detail.
> I
> > > > really like the design, it is a nice mixture of concerns. The result
> of
> > > > PARSE_XML looks and feels like a VARIANT while the XML content is
> > > > preserved in a lossless fashion and all attributes and text is
> > > > accessible. It is a nice combination of what other vendors offer
> (modulo
> > > > the historical legacy they have to deal with).
> > > >
> > > > Looking at various UDF implementations, XML handling is by far the
> most
> > > > important reason for the need of a custom function. So +1 for this
> FLIP.
> > > >
> > > > Thanks,
> > > > Timo
> > > >
> > > > On 11.09.26 14:54, Moritz Manner wrote:
> > > > > Hey everyone,
> > > > >
> > > > > I'd like to start a discussion on FLIP-612: Native XML Functions
> for
> > > > Flink
> > > > > SQL [1].
> > > > >
> > > > > The goal is native XML parsing in Flink SQL, parsing XML into a
> VARIANT
> > > > > instead of requiring custom UDFs.
> > > > >
> > > > > The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML
> turn an
> > > > XML
> > > > > string into a VARIANT, and XML_STRING turns a VARIANT back into an
> XML
> > > > > string.
> > > > > This mirrors the existing JSON functions PARSE_JSON,
> TRY_PARSE_JSON,
> > > and
> > > > > JSON_STRING.
> > > > >
> > > > > Once XML is a VARIANT, you query it the same way as the output of
> > > > PARSE_JSON:
> > > > > with the variant accessors (variant.key, variant['key'],
> > > variant[index])
> > > > > and a CAST to the type you need.
> > > > >
> > > > > The main part is the XML→VARIANT mapping: attributes and text go
> into
> > > @/$
> > > > > fields, and since Variant objects store fields sorted by key rather
> > > than
> > > > in
> > > > > document order, a # field is used to recover the original order
> between
> > > > > differently-named siblings. Details and examples are in the FLIP.
> > > > >
> > > > > It's a small, additive API, no new SQL grammar or Table API methods
> > > > needed.
> > > > >
> > > > > Looking forward to your thoughts!
> > > > >
> > > > > Best,
> > > > > Moritz
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$
> > > >
> > > >
> > >
>
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Krishna Sai <sa...@gmail.com>.
Hi Moritz,
Nice FLIP. The # field is a neat solution, and I checked on master that
Variant does sort object keys, so it is doing necessary work.
One small question: what does PARSE_XML do with an xmlns declaration in
the input? The parser is not namespace-aware, so xmlns:xsi is an
attribute, and by the @name rule it would land as "@xmlns:xsi". If
XML_STRING writes that back and also adds its own xmlns:xsi declaration
for xsi:nil, the root would carry that attribute twice, which is not
well-formed, so that Variant would not round trip.
<root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<a xsi:nil="true"/>
</root>
Dropping xmlns at parse time would instead leave other prefixes
undeclared. Maybe I am misreading the mapping, but a line either way
might help.
Best,
Sai Krishna Sepuri
On Thu, Sep 17, 2026 at 7:51 PM Moritz Manner <mo...@gmail.com> wrote:
>
> Hi Fabian,
>
> Good point, and I agree this is a real problem. A consistent VARIANT
> structure across documents with the same schema is important, especially
> for streaming SQL where you don't want CASE WHEN everywhere, or a job
> failing because it tries to access an array that doesn't exist. Initially,
> the structure optimized for queryability, i.e. a single element stays a
> single node instead of an array. But I think consistency should probably
> win here.
>
> The idea would be to make ARRAY the default for all elements, so the
> structure stays the same no matter how many elements are actually there.
> The downside is that for simple XMLs, where most elements only ever appear
> once, you'd need to add [1] everywhere just to access them.
>
> One option would be to start with ARRAY by default and see how it goes
> based on feedback. If it turns out to be too cumbersome for simple XMLs, we
> could add an opt-out parameter later, that would be a backward-compatible
> addition so it doesn't need to be solved now.
> Does this make sense, or is there a better way to approach this?
>
> Best,
> Moritz
>
> On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev <de...@flink.apache.org>
> wrote:
>
> > Hi everyone and thanks for the proposal Moritz,
> >
> > The FLIP reads very well and I agree (mostly) with the design.
> >
> > I have only one concern, which is about the representation of repeated
> > elements as arrays.
> > I think it is very important that a batch of XML documents with the same
> > structure converts into an identical VARIANT structure.
> > If I understand the current proposal correctly, this would not be the case
> > for two documents: one with a single element and one with a repeated
> > element.
> > The first document would have a single node, while the second document
> > would have an array.
> > If you want to access the nodes, you would need a cumbersome CASE WHEN
> > construct to handle both cases.
> > It would be better to convert both documents into a VARIANT containing an
> > ARRAY (one only holding a single element).
> >
> > Since deriving the right conversion strategy from a single XML document
> > isn't possible, could we add an optional argument to pass conversion hints?
> > There might be other solutions to this problem.
> >
> > IMO, it would also be fine to continue with this FLIP and address this
> > issue later, but having a proposal for it would be good.
> > Do you have any thoughts on this, Moritz?
> >
> > Best, Fabian
> >
> >
> > On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <tw...@apache.org> wrote:
> >
> > > Hi Moritz,
> > >
> > > this is an excellent design document with the right level of detail. I
> > > really like the design, it is a nice mixture of concerns. The result of
> > > PARSE_XML looks and feels like a VARIANT while the XML content is
> > > preserved in a lossless fashion and all attributes and text is
> > > accessible. It is a nice combination of what other vendors offer (modulo
> > > the historical legacy they have to deal with).
> > >
> > > Looking at various UDF implementations, XML handling is by far the most
> > > important reason for the need of a custom function. So +1 for this FLIP.
> > >
> > > Thanks,
> > > Timo
> > >
> > > On 11.09.26 14:54, Moritz Manner wrote:
> > > > Hey everyone,
> > > >
> > > > I'd like to start a discussion on FLIP-612: Native XML Functions for
> > > Flink
> > > > SQL [1].
> > > >
> > > > The goal is native XML parsing in Flink SQL, parsing XML into a VARIANT
> > > > instead of requiring custom UDFs.
> > > >
> > > > The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML turn an
> > > XML
> > > > string into a VARIANT, and XML_STRING turns a VARIANT back into an XML
> > > > string.
> > > > This mirrors the existing JSON functions PARSE_JSON, TRY_PARSE_JSON,
> > and
> > > > JSON_STRING.
> > > >
> > > > Once XML is a VARIANT, you query it the same way as the output of
> > > PARSE_JSON:
> > > > with the variant accessors (variant.key, variant['key'],
> > variant[index])
> > > > and a CAST to the type you need.
> > > >
> > > > The main part is the XML→VARIANT mapping: attributes and text go into
> > @/$
> > > > fields, and since Variant objects store fields sorted by key rather
> > than
> > > in
> > > > document order, a # field is used to recover the original order between
> > > > differently-named siblings. Details and examples are in the FLIP.
> > > >
> > > > It's a small, additive API, no new SQL grammar or Table API methods
> > > needed.
> > > >
> > > > Looking forward to your thoughts!
> > > >
> > > > Best,
> > > > Moritz
> > > >
> > > > [1]
> > > >
> > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$
> > >
> > >
> >
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Moritz Manner <mo...@gmail.com>.
Hi Fabian,
Good point, and I agree this is a real problem. A consistent VARIANT
structure across documents with the same schema is important, especially
for streaming SQL where you don't want CASE WHEN everywhere, or a job
failing because it tries to access an array that doesn't exist. Initially,
the structure optimized for queryability, i.e. a single element stays a
single node instead of an array. But I think consistency should probably
win here.
The idea would be to make ARRAY the default for all elements, so the
structure stays the same no matter how many elements are actually there.
The downside is that for simple XMLs, where most elements only ever appear
once, you'd need to add [1] everywhere just to access them.
One option would be to start with ARRAY by default and see how it goes
based on feedback. If it turns out to be too cumbersome for simple XMLs, we
could add an opt-out parameter later, that would be a backward-compatible
addition so it doesn't need to be solved now.
Does this make sense, or is there a better way to approach this?
Best,
Moritz
On Thu, 17 Sept 2026 at 15:07, Fabian Hüske via dev <de...@flink.apache.org>
wrote:
> Hi everyone and thanks for the proposal Moritz,
>
> The FLIP reads very well and I agree (mostly) with the design.
>
> I have only one concern, which is about the representation of repeated
> elements as arrays.
> I think it is very important that a batch of XML documents with the same
> structure converts into an identical VARIANT structure.
> If I understand the current proposal correctly, this would not be the case
> for two documents: one with a single element and one with a repeated
> element.
> The first document would have a single node, while the second document
> would have an array.
> If you want to access the nodes, you would need a cumbersome CASE WHEN
> construct to handle both cases.
> It would be better to convert both documents into a VARIANT containing an
> ARRAY (one only holding a single element).
>
> Since deriving the right conversion strategy from a single XML document
> isn't possible, could we add an optional argument to pass conversion hints?
> There might be other solutions to this problem.
>
> IMO, it would also be fine to continue with this FLIP and address this
> issue later, but having a proposal for it would be good.
> Do you have any thoughts on this, Moritz?
>
> Best, Fabian
>
>
> On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <tw...@apache.org> wrote:
>
> > Hi Moritz,
> >
> > this is an excellent design document with the right level of detail. I
> > really like the design, it is a nice mixture of concerns. The result of
> > PARSE_XML looks and feels like a VARIANT while the XML content is
> > preserved in a lossless fashion and all attributes and text is
> > accessible. It is a nice combination of what other vendors offer (modulo
> > the historical legacy they have to deal with).
> >
> > Looking at various UDF implementations, XML handling is by far the most
> > important reason for the need of a custom function. So +1 for this FLIP.
> >
> > Thanks,
> > Timo
> >
> > On 11.09.26 14:54, Moritz Manner wrote:
> > > Hey everyone,
> > >
> > > I'd like to start a discussion on FLIP-612: Native XML Functions for
> > Flink
> > > SQL [1].
> > >
> > > The goal is native XML parsing in Flink SQL, parsing XML into a VARIANT
> > > instead of requiring custom UDFs.
> > >
> > > The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML turn an
> > XML
> > > string into a VARIANT, and XML_STRING turns a VARIANT back into an XML
> > > string.
> > > This mirrors the existing JSON functions PARSE_JSON, TRY_PARSE_JSON,
> and
> > > JSON_STRING.
> > >
> > > Once XML is a VARIANT, you query it the same way as the output of
> > PARSE_JSON:
> > > with the variant accessors (variant.key, variant['key'],
> variant[index])
> > > and a CAST to the type you need.
> > >
> > > The main part is the XML→VARIANT mapping: attributes and text go into
> @/$
> > > fields, and since Variant objects store fields sorted by key rather
> than
> > in
> > > document order, a # field is used to recover the original order between
> > > differently-named siblings. Details and examples are in the FLIP.
> > >
> > > It's a small, additive API, no new SQL grammar or Table API methods
> > needed.
> > >
> > > Looking forward to your thoughts!
> > >
> > > Best,
> > > Moritz
> > >
> > > [1]
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$
> >
> >
>
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Fabian Hüske via dev <de...@flink.apache.org>.
Hi everyone and thanks for the proposal Moritz,
The FLIP reads very well and I agree (mostly) with the design.
I have only one concern, which is about the representation of repeated
elements as arrays.
I think it is very important that a batch of XML documents with the same
structure converts into an identical VARIANT structure.
If I understand the current proposal correctly, this would not be the case
for two documents: one with a single element and one with a repeated
element.
The first document would have a single node, while the second document
would have an array.
If you want to access the nodes, you would need a cumbersome CASE WHEN
construct to handle both cases.
It would be better to convert both documents into a VARIANT containing an
ARRAY (one only holding a single element).
Since deriving the right conversion strategy from a single XML document
isn't possible, could we add an optional argument to pass conversion hints?
There might be other solutions to this problem.
IMO, it would also be fine to continue with this FLIP and address this
issue later, but having a proposal for it would be good.
Do you have any thoughts on this, Moritz?
Best, Fabian
On Mon, Sep 14, 2026 at 11:37 AM Timo Walther <tw...@apache.org> wrote:
> Hi Moritz,
>
> this is an excellent design document with the right level of detail. I
> really like the design, it is a nice mixture of concerns. The result of
> PARSE_XML looks and feels like a VARIANT while the XML content is
> preserved in a lossless fashion and all attributes and text is
> accessible. It is a nice combination of what other vendors offer (modulo
> the historical legacy they have to deal with).
>
> Looking at various UDF implementations, XML handling is by far the most
> important reason for the need of a custom function. So +1 for this FLIP.
>
> Thanks,
> Timo
>
> On 11.09.26 14:54, Moritz Manner wrote:
> > Hey everyone,
> >
> > I'd like to start a discussion on FLIP-612: Native XML Functions for
> Flink
> > SQL [1].
> >
> > The goal is native XML parsing in Flink SQL, parsing XML into a VARIANT
> > instead of requiring custom UDFs.
> >
> > The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML turn an
> XML
> > string into a VARIANT, and XML_STRING turns a VARIANT back into an XML
> > string.
> > This mirrors the existing JSON functions PARSE_JSON, TRY_PARSE_JSON, and
> > JSON_STRING.
> >
> > Once XML is a VARIANT, you query it the same way as the output of
> PARSE_JSON:
> > with the variant accessors (variant.key, variant['key'], variant[index])
> > and a CAST to the type you need.
> >
> > The main part is the XML→VARIANT mapping: attributes and text go into @/$
> > fields, and since Variant objects store fields sorted by key rather than
> in
> > document order, a # field is used to recover the original order between
> > differently-named siblings. Details and examples are in the FLIP.
> >
> > It's a small, additive API, no new SQL grammar or Table API methods
> needed.
> >
> > Looking forward to your thoughts!
> >
> > Best,
> > Moritz
> >
> > [1]
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612*Native*XML*Functions*for*Flink*SQL__;KysrKysr!!Ayb5sqE7!pusFK_SVIXLRWBYFy7nKxC-f5N7Uj1ePbB2T6nFANBnqs-0S_zubmv9UygMYHTQQ1LFe9LUt4tGSVL3srSc$
>
>
Re: [DISCUSS] FLIP-612: Native XML Functions for Flink SQL
Posted by Timo Walther <tw...@apache.org>.
Hi Moritz,
this is an excellent design document with the right level of detail. I
really like the design, it is a nice mixture of concerns. The result of
PARSE_XML looks and feels like a VARIANT while the XML content is
preserved in a lossless fashion and all attributes and text is
accessible. It is a nice combination of what other vendors offer (modulo
the historical legacy they have to deal with).
Looking at various UDF implementations, XML handling is by far the most
important reason for the need of a custom function. So +1 for this FLIP.
Thanks,
Timo
On 11.09.26 14:54, Moritz Manner wrote:
> Hey everyone,
>
> I'd like to start a discussion on FLIP-612: Native XML Functions for Flink
> SQL [1].
>
> The goal is native XML parsing in Flink SQL, parsing XML into a VARIANT
> instead of requiring custom UDFs.
>
> The FLIP proposes three functions. PARSE_XML and TRY_PARSE_XML turn an XML
> string into a VARIANT, and XML_STRING turns a VARIANT back into an XML
> string.
> This mirrors the existing JSON functions PARSE_JSON, TRY_PARSE_JSON, and
> JSON_STRING.
>
> Once XML is a VARIANT, you query it the same way as the output of PARSE_JSON:
> with the variant accessors (variant.key, variant['key'], variant[index])
> and a CAST to the type you need.
>
> The main part is the XML→VARIANT mapping: attributes and text go into @/$
> fields, and since Variant objects store fields sorted by key rather than in
> document order, a # field is used to recover the original order between
> differently-named siblings. Details and examples are in the FLIP.
>
> It's a small, additive API, no new SQL grammar or Table API methods needed.
>
> Looking forward to your thoughts!
>
> Best,
> Moritz
>
> [1]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/451974259/FLIP-612+Native+XML+Functions+for+Flink+SQL