You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Dawid Wysakowicz <dw...@apache.org> on 2023/11/15 11:32:37 UTC
[DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Hi,
I would like to propose a follow-up improvement to some of the work that
has been done over the years to the Table API. I posted the proposed
changes in the FLIP-393. I'd like to get to know what others think of
choosing SQL as the serialization format for QueryOperations.
Regards,
Dawid Wysakowicz
[1] https://cwiki.apache.org/confluence/x/vQ2ZE
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by Benchao Li <li...@apache.org>.
The FLIP looks good to me now, let's start the vote.
Dawid Wysakowicz <wy...@gmail.com> 于2023年11月20日周一 22:36写道：
>
> @Benchao I added an example to the page.
>
> If there are no further comments, I'll start a vote on the FLIP tomorrow or
> the next day.
>
> Best,
> Dawid
>
> On Fri, 17 Nov 2023 at 12:20, xiangyu feng <xi...@gmail.com> wrote:
>
> > >After this FLIP is done, FLINK-25015() can utilize this ability to set
> > > job name for queries.
> >
> > +1 for this. Currently, when users submit sql jobs through table api, we
> > can't see the complete SQL string on flink ui. It would be easy for us to
> > finish this feature if we can get serialized sql from QueryOperation
> > directly.
> >
> > So +1 for the overall proposal.
> >
> > Regards,
> > Xiangyu
> >
> > Benchao Li <li...@apache.org> 于2023年11月17日周五 19:07写道：
> >
> > > That sounds good to me, I'm looking forward to it!
> > >
> > > After this FLIP is done, FLINK-25015 can utilize this ability to set
> > > job name for queries.
> > >
> > > Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 21:16写道：
> > > >
> > > > Yes, the idea is to convert the QueryOperation tree into a
> > > > proper/compilable query. To be honest I didn't think it could be done
> > > > differently, sorry if I wasn't clear enough. Yes, it is very much like
> > > > SqlNode#toSqlString you mentioned. I'll add an example of a single
> > > > QueryOperation tree to the FLIP.
> > > >
> > > > I tried to focus only on the public contracts, not on the
> > implementation
> > > > details. I mentioned Expressions, because this requires changing
> > > > semi-public interfaces in BuiltinFunctionDefinitions.
> > > >
> > > > Hope this makes it clearer.
> > > >
> > > > Regards,
> > > > Dawid
> > > >
> > > > On Thu, 16 Nov 2023 at 12:12, Benchao Li <li...@apache.org> wrote:
> > > >
> > > > > Sorry that I wasn't expressing it clearly.
> > > > >
> > > > > Since the FLIP talks about two things: ResolvedExpression and
> > > > > QueryOperation, and you have illustrated how to serialize
> > > > > ResolvedExpression into SQL string. I'm wondering how you'll gonna to
> > > > > convert QueryOperation into SQL string.
> > > > >
> > > > > I was thinking that you proposed to convert the QueryOperation tree
> > > > > into a "complete runnable SQL statement", e.g.
> > > > >
> > > > >
> > >
> > ProjectQueryOperation(x,y)->FilterQueryOperation(z>10)->TableSourceQueryOperation(T),
> > > > > we'll get "SELECT x, y FROM T WHERE z > 10".
> > > > > Maybe I misread it, maybe you just meant to convert each
> > > > > QueryOperation into a row-level SQL string instead the whole tree
> > into
> > > > > a complete SQL statement.
> > > > >
> > > > > The idea of translating whole QueryOperation tree into SQL statement
> > > > > may come from my experience of Apache Calcite, there is a
> > > > > SqlImplementor[1] which convert a RelNode tree into SqlNode, and
> > > > > further we can use SqlNode#toSqlString to unparse it into SQL
> > string.
> > > > > I would assume that most of our QueryOperations are much like the
> > > > > abstraction of Calcite's RelNode, with some exceptions such as
> > > > > PlannerQueryOperation.
> > > > >
> > > > > [1]
> > > > >
> > >
> > https://github.com/apache/calcite/blob/1537555596f8994831ad015af4b9036aa01ebf78/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L141
> > > > >
> > > > > Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四
> > 16:24写道：
> > > > > >
> > > > > > I think the FLIP covers all public contracts that are necessary to
> > be
> > > > > > discussed at that level.
> > > > > >
> > > > > > If you meant you could not find a method that would be called to
> > > trigger
> > > > > > the translation then it is already there. It's just not implemented
> > > yet:
> > > > > > QueryOperation#asSerializableString[1]. As I mentioned this is
> > > mostly a
> > > > > > follow up to previous work.
> > > > > >
> > > > > > Regards,
> > > > > > Dawid
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > >
> > https://github.com/apache/flink/blob/d18a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
> > > > > >
> > > > > > On Wed, 15 Nov 2023 at 16:36, Benchao Li <li...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > +1 for the idea of choosing SQL as the serialization format for
> > > > > > > QueryOperation, thanks for Dawid for driving this FLIP.
> > > > > > >
> > > > > > > Regarding the implementation, I didn't see the proposal for how
> > to
> > > > > > > translate QueryOperation to SQL yet, am I missing something? Or
> > the
> > > > > > > FLIP is still in preliminary state, you just want to gather ideas
> > > > > > > about whether to use SQL or something else as the serialization
> > > format
> > > > > > > for QueryOperation?
> > > > > > >
> > > > > > > Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三
> > 19:34写道：
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > > I would like to propose a follow-up improvement to some of the
> > > work
> > > > > that
> > > > > > > > has been done over the years to the Table API. I posted the
> > > proposed
> > > > > > > > changes in the FLIP-393. I'd like to get to know what others
> > > think of
> > > > > > > > choosing SQL as the serialization format for QueryOperations.
> > > > > > > > Regards,
> > > > > > > > Dawid Wysakowicz
> > > > > > > >
> > > > > > > > [1] https://cwiki.apache.org/confluence/x/vQ2ZE
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best,
> > > > > > > Benchao Li
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Benchao Li
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
> >
--
Best,
Benchao Li
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by Dawid Wysakowicz <wy...@gmail.com>.
@Benchao I added an example to the page.
If there are no further comments, I'll start a vote on the FLIP tomorrow or
the next day.
Best,
Dawid
On Fri, 17 Nov 2023 at 12:20, xiangyu feng <xi...@gmail.com> wrote:
> >After this FLIP is done, FLINK-25015() can utilize this ability to set
> > job name for queries.
>
> +1 for this. Currently, when users submit sql jobs through table api, we
> can't see the complete SQL string on flink ui. It would be easy for us to
> finish this feature if we can get serialized sql from QueryOperation
> directly.
>
> So +1 for the overall proposal.
>
> Regards,
> Xiangyu
>
> Benchao Li <li...@apache.org> 于2023年11月17日周五 19:07写道：
>
> > That sounds good to me, I'm looking forward to it!
> >
> > After this FLIP is done, FLINK-25015 can utilize this ability to set
> > job name for queries.
> >
> > Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 21:16写道：
> > >
> > > Yes, the idea is to convert the QueryOperation tree into a
> > > proper/compilable query. To be honest I didn't think it could be done
> > > differently, sorry if I wasn't clear enough. Yes, it is very much like
> > > SqlNode#toSqlString you mentioned. I'll add an example of a single
> > > QueryOperation tree to the FLIP.
> > >
> > > I tried to focus only on the public contracts, not on the
> implementation
> > > details. I mentioned Expressions, because this requires changing
> > > semi-public interfaces in BuiltinFunctionDefinitions.
> > >
> > > Hope this makes it clearer.
> > >
> > > Regards,
> > > Dawid
> > >
> > > On Thu, 16 Nov 2023 at 12:12, Benchao Li <li...@apache.org> wrote:
> > >
> > > > Sorry that I wasn't expressing it clearly.
> > > >
> > > > Since the FLIP talks about two things: ResolvedExpression and
> > > > QueryOperation, and you have illustrated how to serialize
> > > > ResolvedExpression into SQL string. I'm wondering how you'll gonna to
> > > > convert QueryOperation into SQL string.
> > > >
> > > > I was thinking that you proposed to convert the QueryOperation tree
> > > > into a "complete runnable SQL statement", e.g.
> > > >
> > > >
> >
> ProjectQueryOperation(x,y)->FilterQueryOperation(z>10)->TableSourceQueryOperation(T),
> > > > we'll get "SELECT x, y FROM T WHERE z > 10".
> > > > Maybe I misread it, maybe you just meant to convert each
> > > > QueryOperation into a row-level SQL string instead the whole tree
> into
> > > > a complete SQL statement.
> > > >
> > > > The idea of translating whole QueryOperation tree into SQL statement
> > > > may come from my experience of Apache Calcite, there is a
> > > > SqlImplementor[1] which convert a RelNode tree into SqlNode, and
> > > > further we can use SqlNode#toSqlString to unparse it into SQL
> string.
> > > > I would assume that most of our QueryOperations are much like the
> > > > abstraction of Calcite's RelNode, with some exceptions such as
> > > > PlannerQueryOperation.
> > > >
> > > > [1]
> > > >
> >
> https://github.com/apache/calcite/blob/1537555596f8994831ad015af4b9036aa01ebf78/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L141
> > > >
> > > > Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四
> 16:24写道：
> > > > >
> > > > > I think the FLIP covers all public contracts that are necessary to
> be
> > > > > discussed at that level.
> > > > >
> > > > > If you meant you could not find a method that would be called to
> > trigger
> > > > > the translation then it is already there. It's just not implemented
> > yet:
> > > > > QueryOperation#asSerializableString[1]. As I mentioned this is
> > mostly a
> > > > > follow up to previous work.
> > > > >
> > > > > Regards,
> > > > > Dawid
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://github.com/apache/flink/blob/d18a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
> > > > >
> > > > > On Wed, 15 Nov 2023 at 16:36, Benchao Li <li...@apache.org>
> > wrote:
> > > > >
> > > > > > +1 for the idea of choosing SQL as the serialization format for
> > > > > > QueryOperation, thanks for Dawid for driving this FLIP.
> > > > > >
> > > > > > Regarding the implementation, I didn't see the proposal for how
> to
> > > > > > translate QueryOperation to SQL yet, am I missing something? Or
> the
> > > > > > FLIP is still in preliminary state, you just want to gather ideas
> > > > > > about whether to use SQL or something else as the serialization
> > format
> > > > > > for QueryOperation?
> > > > > >
> > > > > > Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三
> 19:34写道：
> > > > > > >
> > > > > > > Hi,
> > > > > > > I would like to propose a follow-up improvement to some of the
> > work
> > > > that
> > > > > > > has been done over the years to the Table API. I posted the
> > proposed
> > > > > > > changes in the FLIP-393. I'd like to get to know what others
> > think of
> > > > > > > choosing SQL as the serialization format for QueryOperations.
> > > > > > > Regards,
> > > > > > > Dawid Wysakowicz
> > > > > > >
> > > > > > > [1] https://cwiki.apache.org/confluence/x/vQ2ZE
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best,
> > > > > > Benchao Li
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best,
> > > > Benchao Li
> > > >
> >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
>
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by xiangyu feng <xi...@gmail.com>.
>After this FLIP is done, FLINK-25015() can utilize this ability to set
> job name for queries.
+1 for this. Currently, when users submit sql jobs through table api, we
can't see the complete SQL string on flink ui. It would be easy for us to
finish this feature if we can get serialized sql from QueryOperation
directly.
So +1 for the overall proposal.
Regards,
Xiangyu
Benchao Li <li...@apache.org> 于2023年11月17日周五 19:07写道：
> That sounds good to me, I'm looking forward to it!
>
> After this FLIP is done, FLINK-25015 can utilize this ability to set
> job name for queries.
>
> Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 21:16写道：
> >
> > Yes, the idea is to convert the QueryOperation tree into a
> > proper/compilable query. To be honest I didn't think it could be done
> > differently, sorry if I wasn't clear enough. Yes, it is very much like
> > SqlNode#toSqlString you mentioned. I'll add an example of a single
> > QueryOperation tree to the FLIP.
> >
> > I tried to focus only on the public contracts, not on the implementation
> > details. I mentioned Expressions, because this requires changing
> > semi-public interfaces in BuiltinFunctionDefinitions.
> >
> > Hope this makes it clearer.
> >
> > Regards,
> > Dawid
> >
> > On Thu, 16 Nov 2023 at 12:12, Benchao Li <li...@apache.org> wrote:
> >
> > > Sorry that I wasn't expressing it clearly.
> > >
> > > Since the FLIP talks about two things: ResolvedExpression and
> > > QueryOperation, and you have illustrated how to serialize
> > > ResolvedExpression into SQL string. I'm wondering how you'll gonna to
> > > convert QueryOperation into SQL string.
> > >
> > > I was thinking that you proposed to convert the QueryOperation tree
> > > into a "complete runnable SQL statement", e.g.
> > >
> > >
> ProjectQueryOperation(x,y)->FilterQueryOperation(z>10)->TableSourceQueryOperation(T),
> > > we'll get "SELECT x, y FROM T WHERE z > 10".
> > > Maybe I misread it, maybe you just meant to convert each
> > > QueryOperation into a row-level SQL string instead the whole tree into
> > > a complete SQL statement.
> > >
> > > The idea of translating whole QueryOperation tree into SQL statement
> > > may come from my experience of Apache Calcite, there is a
> > > SqlImplementor[1] which convert a RelNode tree into SqlNode, and
> > > further we can use SqlNode#toSqlString to unparse it into SQL string.
> > > I would assume that most of our QueryOperations are much like the
> > > abstraction of Calcite's RelNode, with some exceptions such as
> > > PlannerQueryOperation.
> > >
> > > [1]
> > >
> https://github.com/apache/calcite/blob/1537555596f8994831ad015af4b9036aa01ebf78/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L141
> > >
> > > Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 16:24写道：
> > > >
> > > > I think the FLIP covers all public contracts that are necessary to be
> > > > discussed at that level.
> > > >
> > > > If you meant you could not find a method that would be called to
> trigger
> > > > the translation then it is already there. It's just not implemented
> yet:
> > > > QueryOperation#asSerializableString[1]. As I mentioned this is
> mostly a
> > > > follow up to previous work.
> > > >
> > > > Regards,
> > > > Dawid
> > > >
> > > > [1]
> > > >
> > >
> https://github.com/apache/flink/blob/d18a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
> > > >
> > > > On Wed, 15 Nov 2023 at 16:36, Benchao Li <li...@apache.org>
> wrote:
> > > >
> > > > > +1 for the idea of choosing SQL as the serialization format for
> > > > > QueryOperation, thanks for Dawid for driving this FLIP.
> > > > >
> > > > > Regarding the implementation, I didn't see the proposal for how to
> > > > > translate QueryOperation to SQL yet, am I missing something? Or the
> > > > > FLIP is still in preliminary state, you just want to gather ideas
> > > > > about whether to use SQL or something else as the serialization
> format
> > > > > for QueryOperation?
> > > > >
> > > > > Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三 19:34写道：
> > > > > >
> > > > > > Hi,
> > > > > > I would like to propose a follow-up improvement to some of the
> work
> > > that
> > > > > > has been done over the years to the Table API. I posted the
> proposed
> > > > > > changes in the FLIP-393. I'd like to get to know what others
> think of
> > > > > > choosing SQL as the serialization format for QueryOperations.
> > > > > > Regards,
> > > > > > Dawid Wysakowicz
> > > > > >
> > > > > > [1] https://cwiki.apache.org/confluence/x/vQ2ZE
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best,
> > > > > Benchao Li
> > > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
>
>
>
> --
>
> Best,
> Benchao Li
>
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by Benchao Li <li...@apache.org>.
That sounds good to me, I'm looking forward to it!
After this FLIP is done, FLINK-25015 can utilize this ability to set
job name for queries.
Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 21:16写道：
>
> Yes, the idea is to convert the QueryOperation tree into a
> proper/compilable query. To be honest I didn't think it could be done
> differently, sorry if I wasn't clear enough. Yes, it is very much like
> SqlNode#toSqlString you mentioned. I'll add an example of a single
> QueryOperation tree to the FLIP.
>
> I tried to focus only on the public contracts, not on the implementation
> details. I mentioned Expressions, because this requires changing
> semi-public interfaces in BuiltinFunctionDefinitions.
>
> Hope this makes it clearer.
>
> Regards,
> Dawid
>
> On Thu, 16 Nov 2023 at 12:12, Benchao Li <li...@apache.org> wrote:
>
> > Sorry that I wasn't expressing it clearly.
> >
> > Since the FLIP talks about two things: ResolvedExpression and
> > QueryOperation, and you have illustrated how to serialize
> > ResolvedExpression into SQL string. I'm wondering how you'll gonna to
> > convert QueryOperation into SQL string.
> >
> > I was thinking that you proposed to convert the QueryOperation tree
> > into a "complete runnable SQL statement", e.g.
> >
> > ProjectQueryOperation(x,y)->FilterQueryOperation(z>10)->TableSourceQueryOperation(T),
> > we'll get "SELECT x, y FROM T WHERE z > 10".
> > Maybe I misread it, maybe you just meant to convert each
> > QueryOperation into a row-level SQL string instead the whole tree into
> > a complete SQL statement.
> >
> > The idea of translating whole QueryOperation tree into SQL statement
> > may come from my experience of Apache Calcite, there is a
> > SqlImplementor[1] which convert a RelNode tree into SqlNode, and
> > further we can use SqlNode#toSqlString to unparse it into SQL string.
> > I would assume that most of our QueryOperations are much like the
> > abstraction of Calcite's RelNode, with some exceptions such as
> > PlannerQueryOperation.
> >
> > [1]
> > https://github.com/apache/calcite/blob/1537555596f8994831ad015af4b9036aa01ebf78/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L141
> >
> > Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 16:24写道：
> > >
> > > I think the FLIP covers all public contracts that are necessary to be
> > > discussed at that level.
> > >
> > > If you meant you could not find a method that would be called to trigger
> > > the translation then it is already there. It's just not implemented yet:
> > > QueryOperation#asSerializableString[1]. As I mentioned this is mostly a
> > > follow up to previous work.
> > >
> > > Regards,
> > > Dawid
> > >
> > > [1]
> > >
> > https://github.com/apache/flink/blob/d18a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
> > >
> > > On Wed, 15 Nov 2023 at 16:36, Benchao Li <li...@apache.org> wrote:
> > >
> > > > +1 for the idea of choosing SQL as the serialization format for
> > > > QueryOperation, thanks for Dawid for driving this FLIP.
> > > >
> > > > Regarding the implementation, I didn't see the proposal for how to
> > > > translate QueryOperation to SQL yet, am I missing something? Or the
> > > > FLIP is still in preliminary state, you just want to gather ideas
> > > > about whether to use SQL or something else as the serialization format
> > > > for QueryOperation?
> > > >
> > > > Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三 19:34写道：
> > > > >
> > > > > Hi,
> > > > > I would like to propose a follow-up improvement to some of the work
> > that
> > > > > has been done over the years to the Table API. I posted the proposed
> > > > > changes in the FLIP-393. I'd like to get to know what others think of
> > > > > choosing SQL as the serialization format for QueryOperations.
> > > > > Regards,
> > > > > Dawid Wysakowicz
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/x/vQ2ZE
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best,
> > > > Benchao Li
> > > >
> >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
--
Best,
Benchao Li
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by Dawid Wysakowicz <wy...@gmail.com>.
Yes, the idea is to convert the QueryOperation tree into a
proper/compilable query. To be honest I didn't think it could be done
differently, sorry if I wasn't clear enough. Yes, it is very much like
SqlNode#toSqlString you mentioned. I'll add an example of a single
QueryOperation tree to the FLIP.
I tried to focus only on the public contracts, not on the implementation
details. I mentioned Expressions, because this requires changing
semi-public interfaces in BuiltinFunctionDefinitions.
Hope this makes it clearer.
Regards,
Dawid
On Thu, 16 Nov 2023 at 12:12, Benchao Li <li...@apache.org> wrote:
> Sorry that I wasn't expressing it clearly.
>
> Since the FLIP talks about two things: ResolvedExpression and
> QueryOperation, and you have illustrated how to serialize
> ResolvedExpression into SQL string. I'm wondering how you'll gonna to
> convert QueryOperation into SQL string.
>
> I was thinking that you proposed to convert the QueryOperation tree
> into a "complete runnable SQL statement", e.g.
>
> ProjectQueryOperation(x,y)->FilterQueryOperation(z>10)->TableSourceQueryOperation(T),
> we'll get "SELECT x, y FROM T WHERE z > 10".
> Maybe I misread it, maybe you just meant to convert each
> QueryOperation into a row-level SQL string instead the whole tree into
> a complete SQL statement.
>
> The idea of translating whole QueryOperation tree into SQL statement
> may come from my experience of Apache Calcite, there is a
> SqlImplementor[1] which convert a RelNode tree into SqlNode, and
> further we can use SqlNode#toSqlString to unparse it into SQL string.
> I would assume that most of our QueryOperations are much like the
> abstraction of Calcite's RelNode, with some exceptions such as
> PlannerQueryOperation.
>
> [1]
> https://github.com/apache/calcite/blob/1537555596f8994831ad015af4b9036aa01ebf78/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L141
>
> Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 16:24写道：
> >
> > I think the FLIP covers all public contracts that are necessary to be
> > discussed at that level.
> >
> > If you meant you could not find a method that would be called to trigger
> > the translation then it is already there. It's just not implemented yet:
> > QueryOperation#asSerializableString[1]. As I mentioned this is mostly a
> > follow up to previous work.
> >
> > Regards,
> > Dawid
> >
> > [1]
> >
> https://github.com/apache/flink/blob/d18a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
> >
> > On Wed, 15 Nov 2023 at 16:36, Benchao Li <li...@apache.org> wrote:
> >
> > > +1 for the idea of choosing SQL as the serialization format for
> > > QueryOperation, thanks for Dawid for driving this FLIP.
> > >
> > > Regarding the implementation, I didn't see the proposal for how to
> > > translate QueryOperation to SQL yet, am I missing something? Or the
> > > FLIP is still in preliminary state, you just want to gather ideas
> > > about whether to use SQL or something else as the serialization format
> > > for QueryOperation?
> > >
> > > Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三 19:34写道：
> > > >
> > > > Hi,
> > > > I would like to propose a follow-up improvement to some of the work
> that
> > > > has been done over the years to the Table API. I posted the proposed
> > > > changes in the FLIP-393. I'd like to get to know what others think of
> > > > choosing SQL as the serialization format for QueryOperations.
> > > > Regards,
> > > > Dawid Wysakowicz
> > > >
> > > > [1] https://cwiki.apache.org/confluence/x/vQ2ZE
> > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
>
>
>
> --
>
> Best,
> Benchao Li
>
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by Benchao Li <li...@apache.org>.
Sorry that I wasn't expressing it clearly.
Since the FLIP talks about two things: ResolvedExpression and
QueryOperation, and you have illustrated how to serialize
ResolvedExpression into SQL string. I'm wondering how you'll gonna to
convert QueryOperation into SQL string.
I was thinking that you proposed to convert the QueryOperation tree
into a "complete runnable SQL statement", e.g.
ProjectQueryOperation(x,y)->FilterQueryOperation(z>10)->TableSourceQueryOperation(T),
we'll get "SELECT x, y FROM T WHERE z > 10".
Maybe I misread it, maybe you just meant to convert each
QueryOperation into a row-level SQL string instead the whole tree into
a complete SQL statement.
The idea of translating whole QueryOperation tree into SQL statement
may come from my experience of Apache Calcite, there is a
SqlImplementor[1] which convert a RelNode tree into SqlNode, and
further we can use SqlNode#toSqlString to unparse it into SQL string.
I would assume that most of our QueryOperations are much like the
abstraction of Calcite's RelNode, with some exceptions such as
PlannerQueryOperation.
[1] https://github.com/apache/calcite/blob/1537555596f8994831ad015af4b9036aa01ebf78/core/src/main/java/org/apache/calcite/rel/rel2sql/SqlImplementor.java#L141
Dawid Wysakowicz <wy...@gmail.com> 于2023年11月16日周四 16:24写道：
>
> I think the FLIP covers all public contracts that are necessary to be
> discussed at that level.
>
> If you meant you could not find a method that would be called to trigger
> the translation then it is already there. It's just not implemented yet:
> QueryOperation#asSerializableString[1]. As I mentioned this is mostly a
> follow up to previous work.
>
> Regards,
> Dawid
>
> [1]
> https://github.com/apache/flink/blob/d18a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
>
> On Wed, 15 Nov 2023 at 16:36, Benchao Li <li...@apache.org> wrote:
>
> > +1 for the idea of choosing SQL as the serialization format for
> > QueryOperation, thanks for Dawid for driving this FLIP.
> >
> > Regarding the implementation, I didn't see the proposal for how to
> > translate QueryOperation to SQL yet, am I missing something? Or the
> > FLIP is still in preliminary state, you just want to gather ideas
> > about whether to use SQL or something else as the serialization format
> > for QueryOperation?
> >
> > Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三 19:34写道：
> > >
> > > Hi,
> > > I would like to propose a follow-up improvement to some of the work that
> > > has been done over the years to the Table API. I posted the proposed
> > > changes in the FLIP-393. I'd like to get to know what others think of
> > > choosing SQL as the serialization format for QueryOperations.
> > > Regards,
> > > Dawid Wysakowicz
> > >
> > > [1] https://cwiki.apache.org/confluence/x/vQ2ZE
> >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
--
Best,
Benchao Li
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by Dawid Wysakowicz <wy...@gmail.com>.
I think the FLIP covers all public contracts that are necessary to be
discussed at that level.
If you meant you could not find a method that would be called to trigger
the translation then it is already there. It's just not implemented yet:
QueryOperation#asSerializableString[1]. As I mentioned this is mostly a
follow up to previous work.
Regards,
Dawid
[1]
https://github.com/apache/flink/blob/d18a4bfe596fc580f8280750fa3bfa22007671d9/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/QueryOperation.java#L46
On Wed, 15 Nov 2023 at 16:36, Benchao Li <li...@apache.org> wrote:
> +1 for the idea of choosing SQL as the serialization format for
> QueryOperation, thanks for Dawid for driving this FLIP.
>
> Regarding the implementation, I didn't see the proposal for how to
> translate QueryOperation to SQL yet, am I missing something? Or the
> FLIP is still in preliminary state, you just want to gather ideas
> about whether to use SQL or something else as the serialization format
> for QueryOperation?
>
> Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三 19:34写道：
> >
> > Hi,
> > I would like to propose a follow-up improvement to some of the work that
> > has been done over the years to the Table API. I posted the proposed
> > changes in the FLIP-393. I'd like to get to know what others think of
> > choosing SQL as the serialization format for QueryOperations.
> > Regards,
> > Dawid Wysakowicz
> >
> > [1] https://cwiki.apache.org/confluence/x/vQ2ZE
>
>
>
> --
>
> Best,
> Benchao Li
>
Re: [DISCUSS] FLIP-393: Make QueryOperations SQL serializable
Posted by Benchao Li <li...@apache.org>.
+1 for the idea of choosing SQL as the serialization format for
QueryOperation, thanks for Dawid for driving this FLIP.
Regarding the implementation, I didn't see the proposal for how to
translate QueryOperation to SQL yet, am I missing something? Or the
FLIP is still in preliminary state, you just want to gather ideas
about whether to use SQL or something else as the serialization format
for QueryOperation?
Dawid Wysakowicz <dw...@apache.org> 于2023年11月15日周三 19:34写道：
>
> Hi,
> I would like to propose a follow-up improvement to some of the work that
> has been done over the years to the Table API. I posted the proposed
> changes in the FLIP-393. I'd like to get to know what others think of
> choosing SQL as the serialization format for QueryOperations.
> Regards,
> Dawid Wysakowicz
>
> [1] https://cwiki.apache.org/confluence/x/vQ2ZE
--
Best,
Benchao Li